Chi-square Test

A chi-square test is a statistical method used to determine whether there is a significant association between categorical variables, most commonly applied in A/B testing to compare conversion rates or other binary outcome metrics between variations.

Meaning & Context

The chi-square test compares observed frequencies (actual conversions and non-conversions in each variation) against expected frequencies (what would occur if there were no difference between variations). It produces a test statistic and p-value that indicate whether the observed pattern of results is likely due to the test variation or random chance. This test is ideal for analyzing proportions, percentages, and count data.

Why It Matters

Chi-square tests are the standard statistical method for evaluating A/B tests focused on conversion rates, click-through rates, and other percentage-based metrics. They provide a rigorous framework for decision-making about whether to implement changes based on binary outcomes. Most A/B testing platforms use chi-square tests or similar methods under the hood to calculate statistical significance for conversion metrics.

Example

In testing two different call-to-action buttons, you observe 450 conversions from 10,000 visitors in variation A versus 520 conversions from 10,000 visitors in variation B. A chi-square test determines whether this difference in conversion rates (4.5% vs 5.2%) is statistically significant or could reasonably occur by chance.