Law of Large Numbers

The Law of Large Numbers is a statistical principle stating that as sample size increases, the observed average of results will converge toward the true expected value of the population.

Meaning & Context

In A/B testing, this theorem explains why larger sample sizes produce more reliable and accurate results that better represent actual user behavior. The law guarantees that random fluctuations and outliers have less impact on outcomes as more data is collected. This mathematical principle underpins the requirement for sufficient sample sizes before declaring test winners.

Why It Matters

Understanding this law helps practitioners avoid premature test conclusions based on early data that may not represent true performance differences. It explains why tests need adequate traffic and time to reach statistical significance, preventing costly decisions based on misleading early trends. This principle is fundamental to determining appropriate sample sizes and test duration in experimental design.

Example

In the first 100 visitors to an A/B test, Variation B shows a 50% conversion lift, but after 10,000 visitors, the Law of Large Numbers reveals the true effect is only a 5% improvement as early randomness evens out with larger sample size.