The Replication Problem: Why a 'Winning' A/B Test Sometimes Loses Its Lift
Quick answer
When you pick the winners out of a batch of experiments, the lift you measured is usually a bit better than the lift you'll actually get. Statisticians call this the winner's curse. It's built into how we choose what to ship, which means a test that "won" can look weaker the second time around even when nothing went wrong.
If you've ever shipped a big win and watched the revenue graph shrug, this one's for you.
Key takeaways
- The measured lift on a selected winner is the optimistic end — selection filters keep the results where luck helped most.
- Low-power tests inflate the problem: Gelman & Carlin's Type M (magnitude) error says noisy wins overstate the true effect.
- Carryover from a prior experiment and peeking can both make a "win" look larger than it actually is.
What is the winner's curse?
It starts with a simple filter. We run a test, and if it comes back significant and positive, we launch it. The ones that don't make the cut get shelved.
That filter matters. As Airbnb's Minyong Lee and Milan Shen put it, "usually the ones with significant successful results are chosen to be launched into the product," and they set out to "investigate a statistical selection bias in this process" (Lee & Shen, KDD 2018).
The idea is easy to picture. Every test result is the true effect plus some luck. If you only keep the results that cleared a bar, you're keeping the ones where luck pushed in your favor. Andrea Corvi puts it this way: "the average effect size among wins is larger than the true effect size" (ABTestResult, 2026).
Why does low power make it worse?
The smaller or noisier the test, the bigger the problem.
Statisticians Andrew Gelman and John Carlin call the overstatement a Type M (magnitude) error. They found that "when power gets much below 0.5, the exaggeration ratio becomes high." In noisy settings, they add, significant results "invariably overestimate the absolute values of any actual effect sizes" (Gelman & Carlin, 2014).
That's a different problem from the one most power guides focus on. Low power is usually described as the risk of missing a real lift. Here it works the other way: when a small test does hit significance, the number it reports is more likely to be inflated.
Free A/B Testing Tool
Run your next A/B test the right way
Visual editor, 15 KB script, GA4-native — and free forever up to 100,000 monthly visitors. No developer required.
What else can make a win disappear?
The winner's curse isn't the only culprit. Two others show up in Microsoft's experimentation research.
Early results are noisy. Ron Kohavi and his colleagues showed that in an A/A test, where the true effect is zero, "the first day has a 67% chance of falling outside the 95% confidence bound at the end of the experiment; the second day has a 55% chance." Their conclusion: "early results are often misleading" (Kohavi et al., KDD 2012).
Mida's own guide to peeking describes the same pattern: "what seems like a 'clear winner' may revert to average once the full dataset is collected."
Carryover effects. The same Microsoft paper describes an experiment where surprising results appeared across metrics unrelated to the change. The team reran it on a larger sample and wrote, "many of the effects disappeared." Their explanation wasn't the winner's curse. Users from the earlier experiment were being reused in the follow-on one, so carryover from the first test was skewing the second.
So what should you do about it?
None of this means your wins aren't real. It means the first number you see is a signal, not a forecast. A few habits help. These are suggestions from me, not findings from the papers above:
- Treat the measured lift as the optimistic end. When you build a revenue forecast from a win, plan with something lower. ABTestResult suggests a haircut as a rough planning baseline, though it describes that as its own rule of thumb rather than a law.
- Re-run surprising or high-stakes wins. A result that's much bigger than anything you've seen before deserves a second look before it goes into a roadmap.
- Don't lean on small tests for big claims. If a test barely had the power to detect anything, a significant result from it is the one to be most skeptical of.
- Let the test finish. Stopping the moment a variant pulls ahead is how early noise becomes a "win."
The takeaway
A winning test is real evidence, but it's evidence that's been through a filter. The more you select the best-looking results, the more of your lift is luck, and the more likely a re-run is to come in lower.
That's not a reason to stop shipping winners. It's a reason to set expectations a little below the headline number, and to be most careful with the results that look too good.
Related reading
- What Is Peeking in A/B Testing and Why You Shouldn't Peek!
- How to Calculate Power Statistics for A/B Testing?
- How Long Should I Run My A/B Test?
- Why Your A/B Tests Keep Coming Back Flat
Free A/B Testing Tool
Run your next A/B test the right way
Visual editor, 15 KB script, GA4-native — and free forever up to 100,000 monthly visitors. No developer required.
Sources
- Lee, M., & Shen, M. (2018). Winner's Curse: Bias Estimation for Total Effects of Features in Online Controlled Experiments. KDD 2018.
- Gelman, A., & Carlin, J. (2014). Beyond power calculations: Assessing Type S (sign) and Type M (magnitude) errors. Perspectives on Psychological Science.
- Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T., & Xu, Y. (2012). Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained. KDD '12.
- Corvi, A. (2026, February 5). The Winner's Curse in A/B Testing: Why Wins Overestimate Impact. ABTestResult.
