Frequentist Statistics

Frequentist Statistics is the traditional statistical approach used in A/B testing that determines whether results are significant by calculating the probability of observing the data (or more extreme data) if the null hypothesis were true.

Meaning & Context

This approach treats probability as the long-run frequency of events and relies on p-values, confidence intervals, and fixed significance thresholds (alpha) to make decisions. Frequentist methods require pre-determined sample sizes and don't incorporate prior beliefs into the analysis. The methodology assumes that with infinite repetitions of an experiment, the true effect would be captured within the confidence interval a certain percentage of the time.

Why It Matters

Frequentist statistics remains the most widely used and accepted approach in A/B testing, providing a standardized framework that's well-understood across industries and regulatory bodies. It offers strong protection against false positives when proper procedures are followed, including avoiding peeking at results before reaching the predetermined sample size. Understanding frequentist methods is essential for designing rigorous tests, interpreting results from most A/B testing platforms, and communicating findings credibly.

Example

You design a frequentist A/B test requiring 40,000 visitors per variation, set alpha at 0.05, and commit to not analyzing results until reaching that sample size. After collecting the data, you calculate a p-value of 0.02, leading you to reject the null hypothesis and conclude the treatment significantly outperformed the control.