Simpson's Paradox

Last updated 28 October 2026

Quick answer

Simpson's paradox is a reversal that appears when rates are pooled. A variant can be ahead inside every segment and behind once those segments are added together, because the segments have different base rates and the arms contain different mixes of them.

Key takeaways

A rate can go up inside every segment and still go down in the total, when the mix of segments differs between arms.
In a randomized experiment the mix should be similar. A reversal is a reason to check weights and sample ratio mismatch, not to average harder.
The decision belongs to the segments you randomized for, or to a pre-registered weighted average, not to whichever table looks better after the fact.

Definition

What Simpson's Paradox means in A/B testing

The arithmetic needs two ingredients. The segments must have different outcome rates, and the arms must contain those segments in different proportions. The high-rate segment then dominates whichever arm has more of it, and that mix effect can overwhelm a small within-segment gain. The pattern was described by Edward Simpson in 1951 and shows up whenever someone averages ratios instead of comparing like with like. Pooling two experiments, or pooling devices when one variant was mostly seen on mobile, is the usual testing version. A confounding variable is the general name for the mix factor. Simpson's paradox is what the rates do when you ignore it.

Why Simpson's Paradox matters

The shipped decision is a single number unless you said otherwise. If that number points the wrong way, you will roll out a variant that is worse for every group you actually serve, or you will kill a variant that is better for all of them. Randomized assignment is supposed to keep the mix aligned. When the pooled rate and the segment rates disagree, either the mix slipped, two populations were combined, or both. That is also when a sample ratio mismatch check belongs in the readout, because a bad split is one way the mix stops matching.

Example of Simpson's Paradox

A signup test is split by device. The counts below are a worked example, not a customer export. Mobile control: 1,000 visitors, 2% conversion, 20 signups. Mobile variant: 9,000 visitors, 3% conversion, 270 signups. The variant is ahead on mobile. Desktop control: 9,000 visitors, 10% conversion, 900 signups. Desktop variant: 1,000 visitors, 11% conversion, 110 signups. The variant is ahead on desktop. Pooled control: 920 signups out of 10,000 visitors, 9.2%. Pooled variant: 380 signups out of 10,000 visitors, 3.8%. The control is ahead overall. The variant won both rows and lost the total because it was given almost all of the low-converting mobile traffic, while the control was given almost all of the desktop traffic. A 50/50 randomization would not produce a 9-to-1 device split this large. Treat the pooled 3.8% as an artifact, check the assignment, and do not ship from the total. If the segment gaps had been the pre-registered question, both of those gaps favor the variant, by 1 percentage point each.

How to use Simpson's Paradox

Before the test, name the segments you will report, such as device or new versus returning, and name whether the decision uses the pooled rate or a mix-weighted rate. At the end, compute the rate inside each segment and the share of each segment in each arm. If the segment winners and the pooled winner disagree, stop and reconcile the weights. Weight each segment's effect to the same mix, for example each device's share of total traffic, and report that weighted effect next to the raw pool. Do not drop the segment that "causes" the paradox after seeing that the drop changes the winner.

Common mistake

A common mistake is publishing the pooled rate because it is one number stakeholders can hold, and filing the segment table as an appendix. The appendix is the result when the mixes differ. Another is "fixing" the paradox by deleting the segment with the awkward baseline. That changes the population, which is a different experiment, not a cleaned-up version of this one.

Related A/B testing terms

FAQ

Which table do I ship from, the segments or the total?

The one you named before the test. If the product is used on mobile and desktop and you randomized everyone together, a pre-registered overall rate is the primary result, and a reversal means that overall rate is being driven by mix. If you always decide by device, the segment rates are primary and the pooled rate is a curiosity. Picking the table that favors the variant after you have seen both is not a decision rule.

Does a healthy randomization make Simpson's paradox impossible?

It makes a large mix imbalance unlikely, which is why a dramatic reversal should trigger a sample-ratio check and a look at whether two different experiments were pasted together. It does not make the arithmetic impossible. Different baselines and a modest leftover imbalance can still tilt a pooled rate, especially when one segment converts much higher than the other.

Is this the same thing as a confounding variable?

The mix variable is a confounder of the pooled comparison. It is associated with the arm, because the arms ended up with different mixes, and it is associated with the outcome, because the segments have different baseline rates. Naming it is the start. You still have to recompute the rates inside the segments, or weight the segments to a common mix, before you know the direction of the effect.