Type II Error

Type II Error is a false negative result that occurs when an A/B test fails to detect a real difference between variations, incorrectly concluding there is no significant effect when one actually exists.

Meaning & Context

Also known as a false negative or beta error, this mistake happens when you fail to reject the null hypothesis even though the alternative hypothesis is true. In A/B testing, this means missing out on a genuinely better variation because your test didn't have enough statistical power to detect the difference. The probability of making a Type II Error is represented by beta (β), and statistical power equals 1 - β.

Why It Matters

Type II Errors cause you to miss valuable optimization opportunities, leaving potential revenue and conversions on the table. This often results from insufficient sample sizes, too-short test durations, or testing variations with effects too small to detect reliably. Minimizing Type II Errors requires proper test planning, including power analysis to determine adequate sample sizes before launching tests.

Example

You test a new landing page design that would actually increase conversions by 8%, but your test runs with too small a sample size and concludes 'no significant difference.' You keep the inferior original page, unknowingly sacrificing potential revenue gains.