T-test

A t-test is a statistical hypothesis test used to determine whether there is a significant difference between the means of two groups, commonly applied in A/B testing to compare average metrics like revenue per user or time on site.

Meaning & Context

The t-test calculates a t-statistic that measures the ratio of the difference between group means to the variability within the groups. It produces a p-value that indicates the probability of observing such a difference if no real effect exists. T-tests are particularly appropriate for continuous metrics with normally distributed data and are most reliable with adequate sample sizes.

Why It Matters

T-tests enable experimenters to make statistically rigorous decisions about whether observed differences in continuous metrics are real or simply due to random chance. They're essential for analyzing metrics beyond simple conversion rates, such as average order value, session duration, or pages per visit. Understanding when and how to apply t-tests helps ensure accurate interpretation of test results for revenue and engagement metrics.

Example

After running an A/B test on your checkout page redesign, you use a t-test to analyze whether the average order value of $87.50 in the variation is significantly higher than the $82.30 in the control, determining with 95% confidence that the increase is statistically significant and not due to random variation.