Expected Loss

Expected loss is the average amount of value (revenue, conversions, or other metrics) you would lose by choosing a particular variation if it turns out to be inferior, calculated by integrating the loss function over the posterior probability distribution.

Meaning & Context

Expected loss represents the risk associated with each possible decision in an A/B test, weighted by the probability of each outcome. It's calculated separately for the decision to implement each variation, accounting for all scenarios in which that choice could be wrong and their associated costs. When the expected loss of choosing the best-performing variation becomes acceptably small, you have sufficient evidence to conclude the test.

Why It Matters

Expected loss provides a practical, business-oriented stopping criterion for A/B tests that's more meaningful than p-values or confidence levels. It directly answers the question 'how much could we lose by making this decision now?' enabling teams to balance the cost of uncertainty against the cost of delayed implementation. Using expected loss thresholds aligned with business tolerance for risk leads to more efficient testing and better ROI from experimentation programs.

Example

Your test shows variation B leading with a 2.5% conversion rate versus control's 2.3%, but the expected loss of choosing B is still $3,500 per week, exceeding your $1,000 risk threshold. You continue the test until more data reduces the expected loss to an acceptable level before implementing the change.