Statistical Significance

Statistical Significance is the determination that an observed difference between test variations is unlikely to have occurred by chance alone, typically indicated when the p-value falls below the predetermined alpha threshold.

Meaning & Context

When a result is statistically significant (commonly at p < 0.05), it means there's strong evidence against the null hypothesis, suggesting a real effect exists. However, statistical significance doesn't indicate the size or practical importance of the effect—it only addresses whether the difference is likely real rather than random. Significance depends on effect size, sample size, and variance; with large enough samples, even trivially small differences can become statistically significant.

Why It Matters

Statistical significance provides a standardized framework for deciding whether to implement test variations, helping prevent decisions based on random noise. However, it's crucial to consider both statistical significance and practical significance (effect size) together—a statistically significant 0.5% lift might not justify implementation costs. Misunderstanding statistical significance is a common pitfall; it doesn't tell you the probability that your hypothesis is true, nor does it guarantee the effect will persist after implementation.

Example

Your A/B test shows that Variant B increased conversions from 5.2% to 5.8% with p = 0.03. This result is statistically significant at alpha = 0.05, giving you confidence the 0.6 percentage point improvement is real. However, you still need to evaluate whether this 11.5% relative lift justifies the development costs.