Null Hypothesis Significance Testing

Null Hypothesis Significance Testing (NHST) is a statistical method used to determine whether observed differences between test variations are statistically significant or likely due to random chance. It involves testing a null hypothesis that assumes no difference exists between variations against an alternative hypothesis that a difference does exist.

Meaning & Context

In A/B testing, NHST begins with the assumption that both variants perform identically (the null hypothesis). Statistical tests calculate the probability (p-value) of observing the measured difference if the null hypothesis were true. If this probability falls below a predetermined threshold (typically 0.05), the null hypothesis is rejected, suggesting a statistically significant difference exists.

Why It Matters

NHST provides the mathematical foundation for determining whether A/B test results are reliable or merely random fluctuations. It helps prevent false conclusions by quantifying the likelihood that observed performance differences are genuine, enabling data-driven decisions about which variation to implement. Without NHST, experimenters risk making costly changes based on statistical noise.

Example

If Variation B shows a 3% conversion rate increase over Control A, NHST helps determine whether this 3% lift is a real improvement or could have occurred by chance. A p-value of 0.02 would indicate only a 2% probability the difference is random, supporting the conclusion that Variation B truly performs better.