Subjective Probability

Subjective probability is a Bayesian interpretation of probability that represents an individual's degree of belief or confidence about an uncertain event, based on available evidence and prior knowledge. Unlike frequentist probability, it treats probability as a measure of personal certainty rather than long-run frequency.

Meaning & Context

In Bayesian A/B testing approaches, subjective probability allows experimenters to incorporate prior beliefs or historical data into their analysis. This framework updates initial probability assessments as new test data accumulates, producing statements like "there's an 85% probability that Variation B is better than Control A." The subjective nature doesn't mean arbitrary; it's grounded in mathematical principles and available evidence.

Why It Matters

Subjective probability enables more intuitive interpretation of A/B test results compared to frequentist methods. It allows statements about the probability that one variation beats another, which directly answers business questions decision-makers ask. This approach is particularly valuable when incorporating historical performance data or expert knowledge into test analysis.

Example

Using Bayesian analysis, an A/B testing platform might report "there is a 92% probability that the new checkout flow increases conversions," directly expressing confidence in the outcome. This is more intuitive than a frequentist p-value, which only tells you the probability of seeing your data if there were no real difference.