Confounding Variables

Confounding variables are external factors that influence both the independent variable (the change being tested) and the dependent variable (the metric being measured), creating a false or misleading association between them.

Meaning & Context

In A/B testing, confounding variables can corrupt test results by introducing bias that makes it appear one variation is performing better or worse than it actually is. These variables are not part of the intended test design but affect outcomes nonetheless. Common confounding variables include seasonality, traffic source changes, browser updates, or marketing campaigns running simultaneously with the test.

Why It Matters

Uncontrolled confounding variables can lead to incorrect conclusions and poor business decisions based on flawed test results. Proper randomization and controlled testing environments help minimize their impact. Identifying and accounting for potential confounders is essential for ensuring test validity and making reliable optimization decisions.

Example

If you launch an A/B test on the same day your company starts a major TV advertising campaign, the increased traffic and brand awareness from the ads could be a confounding variable, making it impossible to determine whether conversion rate improvements came from your test variation or from the advertising boost.