Test Collisions: What Happens When Two A/B Tests Run on the Same Page at Once

What exactly is a "test collision"?

Picture two tests running at once: one changing your hero headline, another changing the CTA button color directly below it. Individually, each might have a clean, measurable effect. Together, they can produce something different — a bolder headline might make the new button color pop in a way the old one never would have, or clash with it. As Harness's engineering team puts it, "interaction effects occur when two experiments modify the same page element or user flow, creating combined impacts that differ from isolated effects." Neither test's dashboard will tell you this is happening — both will just report numbers, and the numbers will be wrong by an amount neither team can see from inside their own results.

Is this a real risk, or an edge case for companies running thousands of tests?

It's tempting to assume this is a Big Tech problem. Statsig data scientist Kane Luo notes that "a study by Microsoft shows that interaction effects are rare" — but that's rare given Microsoft's infrastructure, which includes dedicated tooling most teams don't have. Bing's experimentation platform, for instance, "runs a daily job that tests each pair of experiments for additivity of their treatment effects" across hundreds of concurrent tests (Kohavi et al., 2013). A team without that safety net isn't automatically safer — it just can't see the problem.

Mida's own test prioritization guide puts a number on it for teams without that infrastructure: "there's a real ceiling on how many concurrent tests a single team can run before interaction effects start corrupting the data — for many teams it's somewhere in the 2–4 range." That's a much lower bar than "hundreds," and it's the number that actually applies to most CRO teams. Even hyperscale companies take the risk seriously enough to build real infrastructure around it — LinkedIn, for example, runs paired "two-sided randomization" experiments to separate production-side and consumption-side changes specifically to avoid this kind of contamination (Gupta et al., 2019).

How do you actually tell if two tests are contaminating each other?

At platform scale, the answer is statistical: "regression models with interaction terms are your friend here… you can actually quantify whether Test A is messing with Test B's results" (Statsig, 2025). Statsig's own detection method splits exposed users into the four combinations of both tests' variants and checks whether the differences between groups are larger than chance would produce (Luo, 2025) — the same four-group logic Harness recommends teams check manually when they suspect a collision.

You don't need a dedicated data science team to use a rougher version of this. If a test that had been flat for weeks suddenly moves the same week another test ships nearby, or a result reverses direction right after an unrelated test ends, that's the practical version of the same signal: it's disagreeing with itself in a way the single-test model can't explain.

What do you actually do about it?

There are two real approaches to prevention, and they trade off against each other. Mutually exclusive experiment groups are the simple option — Statsig calls them "your first line of defense: each user gets assigned to exactly one test" — but they cost you sample size and testing throughput, since every experiment is competing for the same pool of users. The more scalable option is what Google and Microsoft call "numberlines" or "layers": infrastructure that guarantees experiments on the same layer get an exclusive random sample, so unrelated tests can run in parallel without touching each other (Gupta et al., 2019). That's a real engineering investment most CRO teams won't build for a handful of concurrent tests, which is exactly why Mida's own guidance on server-side vs. client-side testing recommends a lighter middle ground: a shared experiment registry so every team can see what's live and where, before two tests ever land on the same element.

Whichever you pick, the one option statistician John D. Cook warns against is picking none: "if you only test one factor at a time, you're betting that interaction effects don't matter." That bet is invisible right up until it isn't.

The takeaway

A test collision isn't a rare, exotic failure mode reserved for companies running thousands of experiments — it's a structural risk of stacking any two tests on the same page or flow, and the "2–4 concurrent tests" ceiling means most CRO teams are closer to this problem than they think. You don't need Bing's daily interaction-detection job to manage it. A shared registry of what's currently live, mutual exclusivity where it's cheap to enforce, and a habit of double-checking any result that moves right when another test ships, will catch the overwhelming majority of collisions before they quietly cost you a decision.

Related reading

• Multivariate Testing vs A/B Testing: When to Use Each (With Real Examples)
• Test Prioritization Frameworks Compared: ICE vs PIE vs PXL
• Server-Side vs Client-Side A/B Testing: When to Use Each

Sources

• Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y., & Pohlmann, N. (2013). Online Controlled Experiments at Large Scale. KDD '13.
• Gupta, S., Kohavi, R., Tang, D., Xu, Y., et al. (2019). Top Challenges from the First Practical Online Controlled Experiments Summit. SIGKDD Explorations, 21(1).
• Statsig. (2025, June 23). Experiment contamination: Preventing test cross-pollution.
• Luo, K. (2025, January 13). Detecting interaction effects of concurrent experiments. Statsig.
• Harness. (2021, April 6). Simultaneous Experimentation: Run Multiple A/B Tests.
• Cook, J. D. (2022, October 31). Interaction effects.