Holdout Group

Definition

A holdout group is a randomized set of visitors who stay on the old experience after a winning variant is rolled out to everyone else. The holdout exists so you can see whether the gain survives past the test, including after the novelty effect fades.

What Holdout Group means in A/B testing

During an A/B test the control group is one of the arms, often half the sample, and it exists to estimate the effect. A holdout comes later. You have decided the variant is the new default, but you keep a slice of traffic on the previous default and you keep scoring the same metric. The comparison is no longer "which idea is better under a 50/50 split." It is "now that this is the site, does the old site still lose." Feature flags are a related control: Mida's flag notes describe turning an experience on for part of an audience and measuring that rollout. A holdout is the measurement habit. The flag or the traffic allocation is one way to implement it.

Why Holdout Group matters

Test wins shrink. Seasonality, a later campaign, and simple familiarity all move the number you saw in a two-week test. A holdout is the only group that still shows you the counterfactual after launch. It also catches a rollout mistake, such as the winner applied on top of another experiment that was not in the original test. Without it, a faded win looks like the next test failed, because the baseline you thought you had was the test's optimistic window.

Example of Holdout Group

A checkout headline wins a four-week test: revenue per visitor is $4.80 on the variant and $4.40 on the control, a 9% relative lift, on a healthy 50/50 split. You roll the headline out to 90% of visitors and keep a random 10% on the old headline. Over the next four weeks the 90% group is at $4.62 and the holdout is at $4.45. The gap is smaller than 9% and it is still on the same side, so you keep the headline and you update the expected gain downward. If the two groups had met at $4.45, the original win was the launch window, and the holdout would tell you to revert. These figures are an illustration of the comparison, not a reported Mida result.

How to use Holdout Group

Randomize the holdout with the same unit you used in the test, usually a visitor id, and do not let people cross in and out. Decide the metric and the length before the rollout. The length should cover the novelty window you care about, often several weeks. Read the holdout against the rolled-out group on that metric only. When the holdout has done its job, ship the remainder or revert. Do not quietly add a second change to the 90% group while the holdout is your only clean comparison.

Common mistake

A common mistake is building the holdout from people who were already in the original control. They are a selected, older cohort, not a fresh random slice of today's traffic. Another is calling a 10% group that was never logged a holdout. If you did not record who was withheld, you cannot compute the gap.

How Mida works with a holdout group

Mida's QA checklist separates traffic allocation from the variant split. Allocation is the share of visitors who enter an experience at all. A 50% allocation means the other half never enter. That is the lever for leaving people on the untouched site. The split inside the experience is a different setting. Feature flags are documented for gradual exposure, so a change can be on for part of the audience and attached to a goal. This page does not describe a separate "holdout" button. You get a holdout by keeping a randomized slice on the old experience and still scoring it.

Related A/B testing terms

• Control Group
• Treatment Group
• Randomization
• Testing Period

FAQ

Why not just leave the original test running?

You can, and a long-running 50/50 is a kind of holdout. Teams shrink the control to 10% after they have decided, so most visitors get the winner while a measured slice still sees the old page. The original 50/50 already answered 'is there an effect.' The smaller holdout answers 'does it still exist after launch.'

How big should the holdout be?

Large enough to notice a fade you would act on, and small enough that you can live with withholding the winner. Ten percent is a common illustration, not a rule. If the metric is rare, 10% will not be readable and you should keep a larger slice or a shorter list of metrics.

Who should be excluded from the holdout?

Anyone you cannot ethically or operationally leave behind, such as a bug fix. A holdout is for a change you are choosing to keep because of its measured benefit. It is a poor tool for outages, security patches, or a price you are legally required to show everyone.