Loss Function

A loss function quantifies the cost or negative consequence of making a wrong decision in A/B testing, typically measuring the expected loss in revenue, conversions, or other key metrics that would result from choosing an inferior variation.

Meaning & Context

In Bayesian A/B testing, loss functions formalize the business impact of decision errors by calculating the opportunity cost of selecting the worse variation. Different loss functions can reflect different business priorities, such as maximizing conversions, minimizing downside risk, or optimizing revenue. The expected loss for each decision (implementing A or B) is calculated across the entire posterior distribution.

Why It Matters

Loss functions connect statistical analysis directly to business outcomes, enabling data-driven decisions based on actual costs rather than arbitrary significance thresholds. They help determine when to stop a test by quantifying whether the potential gain from continuing outweighs the certain cost of delayed implementation. Using loss functions allows teams to make economically rational decisions that balance statistical uncertainty against business impact.

Example

In testing two pricing strategies, your loss function calculates that if you incorrectly choose the worse option, you'd lose an expected $15,000 per month in revenue. When the expected loss of choosing either variation drops below your $2,000 threshold, you have sufficient confidence to make a decision.