Test Collisions: What Happens When Two A/B Tests Run on the Same Page at Once

What exactly is a "test collision"?

Picture two tests running at once: one changing your hero headline, another changing the CTA button color directly below it. On their own, each might have a clean, measurable effect. Together, they can do something different. A bolder headline might make the new button color pop in a way the old one never would, or clash with it. As Harness's engineering team puts it, "interaction effects occur when two experiments modify the same page element or user flow, creating combined impacts that differ from isolated effects."

Neither test's results page will tell you this is happening. Both will just report numbers, and the numbers will be off by an amount you can't see from inside either test.

There's a second, sneakier kind of collision: one test changing who reaches the other. If a homepage test sends more people to a product page, the product page test now gets a different mix of visitors in each of its variants. Its split looks random, but it isn't.

Is this a real risk, or an edge case for companies running thousands of tests?

It's tempting to assume this is a Big Tech problem. Statsig data scientist Kane Luo notes that "a study by Microsoft shows that interaction effects are rare". But that's rare given Microsoft's infrastructure, which includes tooling most teams don't have. Bing's experimentation platform, for instance, "runs a daily job that tests each pair of experiments for additivity of their treatment effects" across hundreds of concurrent tests (Kohavi et al., 2013). A team without that safety net isn't automatically safer. It just can't see the problem.

Mida's own test prioritization guide puts a number on the ceiling for teams without that infrastructure: "for many teams it's somewhere in the 2–4 range, though it depends on traffic, page overlap, and how isolated each test is." That's a much lower bar than "hundreds," and it's the number that applies to most CRO teams.

Even hyperscale companies take the risk seriously enough to build real infrastructure around it. LinkedIn, for example, runs paired "two-sided randomization" experiments to separate production-side and consumption-side changes, specifically to avoid this kind of contamination (Gupta et al., 2019).

How do you tell if two tests are contaminating each other?

At platform scale, the answer is statistical: "regression models with interaction terms are your friend here… you can actually quantify whether Test A is messing with Test B's results" (Statsig, 2025). Statsig's detection method splits exposed users into the four combinations of both tests' variants and checks whether the differences between groups are larger than chance would produce (Luo, 2025). Harness recommends the same four-group check when you suspect a collision.

Without that breakdown, you're left with a rougher signal. If a test that had been flat for weeks suddenly moves the same week another test ships nearby, or a result reverses right after an unrelated test ends, the test is disagreeing with itself in a way a single-test readout can't explain.

The four-group check is better, and you no longer need a data scientist to run it.

Two overlapping tests are a multivariate test you didn't plan

Here's the part most collision articles skip. If two A/B tests each split visitors at random, and they run on the same visitors at the same time, every visitor lands in one of four groups:

That's a full-factorial multivariate test. The data is already there. Nobody looks at it because each test's results page only shows its own two variants.

Read the grid and the collision question answers itself. If "both changes" does about as well as you'd expect from the two single changes, the tests aren't interfering and you can trust both results. If it's much better or much worse, they interact, and the grid tells you which combination to ship.

Reading overlapping tests together with Mida's Combined analysis

Combined analysis does that four-group check for you, for 2 or 3 experiments at once.

1. On the Experiments list, tick the 2 or 3 experiments you want to read together.
2. Click Combine Campaigns.
3. Pick the goal. It can be any goal from the selected experiments, including secondary metrics.

You don't always have to go looking. When live experiments with enough traffic run on the same page for the same audience, Mida shows an Analyze these experiments together note on the Experiments list and on the results page, with a link straight to the combined analysis.

What you get looks like a normal Mida results page, with one row per combination:

• Every combination of the experiments' variants, with visitors, conversion rate, lift and credible interval against the baseline (all originals).
• Chance to Beat Baseline and Chance to be Best for each combination, using the same Bayesian method as the results page. Mida only names a best combination once every combination has enough conversions to compare.
• Each experiment among the same visitors: every test's own result, limited to visitors who saw all of them. If a test's result changes a lot here, the other test is affecting it.
• Split Check, which flags when visitors didn't split evenly across the experiments. That's the "one test changes who reaches the other" collision from earlier.
• Filters and segments. Click a variant to show only the combinations that include it, or apply the same segments you use on the results page.

Two rules keep the numbers honest. Only visitors who entered every selected experiment while they were running together are counted, so returning visitors who saw one test last month and the other today don't count as overlap. And conversions are counted exactly as the goal's own experiment counts them, so the totals match its results page.

A few limits worth knowing:

• Each combination gets a slice of the traffic. Two tests with two variants each make four combinations; three tests with three variants each make 27. More combinations means each one needs more time. Mida shows roughly how many more days you need at current traffic.
• The experiments must have run at the same time. If one ended before the other started, Mida tells you there's nothing to combine.
• It works on standard A/B tests in the same project. Multi-armed bandits, multivariate tests, rollouts and personalizations can't be combined.
• Combined analysis results are included on paid plans. Free accounts can set one up, but the results stay locked.

So what do you do about collisions?

There are three real options, and they suit different situations.

Keep the tests apart. Mutually exclusive groups are the simple option. Statsig calls them "your first line of defense: each user gets assigned to exactly one test." In Mida, that's a Mutual Exclusion Group. The cost is traffic: every test in the group competes for the same pool of visitors, so each one takes longer. Use this when two tests change the same element (two headline tests on one hero, for example). There, one variant simply overwrites the other, so some combinations never really exist.

Run them together and read them together. When tests change different things on the same page or flow, letting them overlap keeps your full traffic in every test. Combined analysis then shows you whether they interact and which combination wins. This is the option most teams skip, because until now it meant exporting data and building the grid by hand.

Use layers. Google and Microsoft use "numberlines" or "layers": infrastructure that guarantees experiments on the same layer get an exclusive random sample, so unrelated tests can run in parallel without touching each other (Gupta et al., 2019). That's a real engineering investment most CRO teams won't build for a handful of concurrent tests.

Whichever you pick, keep a shared view of what's live and where, as Mida's guide on server-side vs. client-side testing recommends, so two tests never land on the same element by accident.

The one option statistician John D. Cook warns against is picking none: "if you only test one factor at a time, you're betting that interaction effects don't matter." That bet is invisible right up until it isn't.

The takeaway

A test collision isn't a rare failure mode reserved for companies running thousands of experiments. It's a built-in risk of stacking any two tests on the same page or flow, and most CRO teams are closer to it than they think.

But overlap isn't only a risk. Two randomized tests on the same visitors give you every combination of their variants for free. Keep tests apart when they fight over the same element. When they don't, let them run together and read the combinations. You'll catch the collisions, and you'll often find a combination that beats either change on its own.

Related reading

• Multivariate Testing vs A/B Testing: When to Use Each (With Real Examples)
• Test Prioritization Frameworks Compared: ICE vs PIE vs PXL
• Server-Side vs Client-Side A/B Testing: When to Use Each

Sources

• Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y., & Pohlmann, N. (2013). Online Controlled Experiments at Large Scale. KDD '13.
• Gupta, S., Kohavi, R., Tang, D., Xu, Y., et al. (2019). Top Challenges from the First Practical Online Controlled Experiments Summit. SIGKDD Explorations, 21(1).
• Statsig. (2025, June 23). Experiment contamination: Preventing test cross-pollution.
• Luo, K. (2025, January 13). Detecting interaction effects of concurrent experiments. Statsig.
• Harness. (2021, April 6). Simultaneous Experimentation: Run Multiple A/B Tests.
• Cook, J. D. (2022, October 31). Interaction effects.