Posterior Probability

Posterior probability is the updated probability of a hypothesis being true after taking into account new evidence or data, calculated using Bayesian statistical methods by combining prior beliefs with observed experimental results.

Meaning & Context

In Bayesian A/B testing, the posterior probability represents your refined understanding of which variation is truly better after seeing the test data. It's calculated by updating your prior probability distribution with the likelihood of the observed data using Bayes' theorem. The posterior probability is typically expressed as the probability that variation B beats variation A, providing an intuitive measure for decision-making.

Why It Matters

Posterior probabilities offer a more intuitive interpretation than traditional p-values, directly answering the question 'what's the probability that this variation is actually better?' This makes results easier to communicate to stakeholders and enables better decision-making under uncertainty. Posterior probabilities also allow you to incorporate prior knowledge and make decisions earlier by quantifying the risk of choosing the wrong variation.

Example

After running a Bayesian A/B test for one week, your analysis shows a posterior probability of 94% that the new checkout flow is better than the current one, meaning there's a 94% chance it truly has a higher conversion rate based on the data observed and your prior assumptions.