Sample Ratio Mismatch

Definition

Sample ratio mismatch (SRM) is a failure of assignment or counting: the share of visitors in each variant is too far from the split you configured for chance to be a good explanation. An A/B test with SRM does not have a trustworthy conversion rate, no matter how significant that rate looks.

What Sample Ratio Mismatch means in A/B testing

You choose an allocation before launch, often 50/50, sometimes 90/10 or a three-way split. Randomization should land near those weights. The check is a chi-square goodness-of-fit test. Expected count for an arm is its intended share times the total visitors. The statistic adds up (observed − expected)² / expected across arms. One arm uses a chi-square with degrees of freedom equal to the number of arms minus one. Small samples are a bad fit for this test: if any expected count is under 5, the result is not reliable yet. Mida's sample ratio mismatch calculator runs that test for up to eight variants and for uneven intended splits. It flags SRM when the p-value is below 0.001, and it shows a borderline warning from 0.001 to 0.01. The significance calculator also reports whether a two-arm 50/50 split looks healthy. The dedicated calculator is the one that accepts a non-equal plan.

Why Sample Ratio Mismatch matters

The missing people are almost never missing at random. A redirect variant that loses slow connections removes impatient visitors from one arm. A variant script that throws in one browser never counts those visitors. A cache that keeps serving the control does the same. The conversion rates you compare then come from different populations, so a p-value on the conversion rate does not answer the question you asked. The QA checklist puts an SRM check in the first day and again before the final read.

Example of Sample Ratio Mismatch

A 50/50 test ends with 52,000 visitors on the control and 48,000 on the variant. Total N is 100,000, so each arm expected 50,000. Chi-square is (2,000)² / 50,000 + (2,000)² / 50,000 = 80 + 80 = 160 on 1 degree of freedom. A chi-square of 160 is far below a p-value of 0.001. This is the same 52,000 versus 48,000 gap the calculator page uses as an obvious mismatch. You do not interpret the conversion rates. You look for a redirect, a bot filter, a variant error, a cache, or an audience that was edited mid-test, then you restart from zero.

How to use Sample Ratio Mismatch

Write the intended weights down before the first visitor. Check the split after the first day of real traffic, and again at the end, by entering observed visitors and those weights in the SRM calculator . If an expected count is still under 5, wait. If the calculator flags a mismatch, stop the analysis. Do not reweight the results and ship them.

Common mistake

A common mistake is using a 0.05 cutoff because that is the conversion-rate habit. On a check you run for every test, 0.05 will fail healthy experiments. Another is ignoring a 90/10 test that came back 80/20 because "it was supposed to be uneven." Uneven is fine only when it matches the uneven plan you set.

How Mida works with sample ratio mismatch

The public check is the SRM calculator: observed visitors, intended split, chi-square p-value, a fail under 0.001, and a warning from 0.001 to 0.01. Causes listed with that calculator are a redirect that drops slow connections, a bot filter that hits one arm, variant code that fails in some browsers so those visitors are never counted, a cache or CDN serving one variant, and an audience edited during the test. The QA checklist uses the same idea as a launch check. This page does not claim a separate in-product SRM diagnosis beyond those tools.

Related A/B testing terms

• Chi-square Test
• Randomization
• Sample Size
• P-value

FAQ

Why is the SRM cutoff 0.001 and not 0.05?

You run this check on every experiment, so a 5% false-alarm rate would flag a healthy test constantly. Mida's calculator uses 0.001 as the fail line, which is the stricter bar common in experimentation, and it treats the band from 0.001 to 0.01 as a warning rather than an automatic fail.

A 90/10 test looks uneven. Is that automatically SRM?

Only if you compare it with a 50/50 expectation. Enter the intended weights, such as 90 and 10. They do not have to sum to 100. The calculator normalizes them and tests the observed counts against that ratio.

Can I keep the conversions and drop the extra visitors?

No. The visitors who went missing are rarely a random subset. A redirect that drops slow phones, or a variant script that errors in one browser, removes a particular kind of person from one arm. The conversion rates are then from different populations. Fix the cause and start a new test.