Bayes Theorem

Bayes theorem is a mathematical formula that describes how to update the probability of a hypothesis based on new evidence, forming the foundation of Bayesian A/B testing by combining prior beliefs with observed data to produce posterior probabilities.

Meaning & Context

The theorem states that the posterior probability is proportional to the likelihood of the observed data given the hypothesis multiplied by the prior probability of the hypothesis. In A/B testing, this allows experimenters to continuously update their beliefs about which variation performs better as data accumulates. Bayes theorem provides a principled framework for learning from data while accounting for existing knowledge and uncertainty.

Why It Matters

Bayes theorem enables a more flexible and intuitive approach to A/B testing compared to traditional frequentist methods. It allows for continuous monitoring without multiple testing penalties, earlier decision-making based on probability statements, and natural incorporation of prior knowledge. Understanding Bayes theorem is essential for implementing and interpreting Bayesian experimentation frameworks.

Example

Using Bayes theorem, you start with a prior belief about your email signup form's 15% conversion rate, then as 1,000 test visitors interact with a new variation, you continuously update your probability estimate that the new form is better, refining your posterior distribution with each new data point collected.