Beta

Beta is the probability of making a Type II Error in hypothesis testing, representing the risk of failing to detect a true difference between variations when one actually exists.

Meaning & Context

Beta is inversely related to statistical power, where power = 1 - β. If you set beta at 0.20 (20%), your test has 80% power, meaning an 80% chance of detecting a real effect if it exists. Beta is determined by your sample size, the minimum detectable effect you want to identify, and your alpha level. Most A/B testing best practices recommend aiming for beta ≤ 0.20 (power ≥ 80%).

Why It Matters

Understanding and controlling beta helps you design tests with adequate statistical power to detect meaningful improvements, preventing you from abandoning genuinely better variations due to inconclusive results. Reducing beta requires increasing sample size, which directly impacts test duration and resource allocation. Power analysis using beta calculations should be performed before launching tests to ensure you collect enough data to reach reliable conclusions.

Example

You conduct a power analysis showing you need 50,000 visitors per variation to achieve beta = 0.20 (80% power) for detecting a 5% lift. Running the test with only 10,000 visitors would increase beta significantly, risking that you'll miss a real improvement.