Peeking

Last updated 28 October 2026

Quick answer

Peeking is checking an A/B test before it reaches the sample size or duration you set, and treating an interim result as a reason to stop. It is an analysis mistake, not a kind of test.

Key takeaways

Peeking means acting on an interim look that was not part of the analysis plan.
Each unplanned look is another chance to mistake noise for a winner, so the real false-positive rate rises above the 5% you set.
The longer walkthrough is the peeking post. A sequential rule has to be chosen before the data, and this site's plan comparison says Mida does not include sequential testing.

Definition

Peeking is checking an A/B test before it reaches the sample size or duration you set, and treating an interim result as a reason to stop. It is an analysis mistake, not a kind of test. The full account is What is peeking in A/B testing?

What Peeking means in A/B testing

A fixed-horizon test picks one analysis, usually at a 5% false-positive limit, after a set number of visitors. That limit applies to one look. If you look on day three, again on day five, and stop the first time the result looks significant, you have given noise several chances to cross the line. The procedure no longer has a 5% error rate. Sequential testing is the method that spends that error rate across planned looks. A boundary invented after a good week is still peeking.

Why Peeking matters

An early lead often shrinks. Shipping it means you roll out a change that was a lucky week, then the next test starts from a baseline that was never real. The QA checklist says not to stop because an interim result looks significant, and to keep the pre-planned sample unless a formal sequential method was chosen in advance. A plan comparison on this site states that sequential testing is not a Mida feature, so the documented rule is the sample you calculated before launch.

Example of Peeking

You need 8,000 visitors per variant. On day two the variant is ahead and the readout shows about 97% significance with thousands of visitors still remaining. Stopping there, because the percentage looks decisive, is peeking. Waiting until both arms reach 8,000, then reading the result once, is the fixed-horizon test you planned. The day-two gap can be the start of a real lift or a streak. One planned look is what tells those apart.

How to use Peeking

Write the sample size and the single readout date before the test starts. Use the sample size calculator for the count and the duration calculator for the days. Do not add a winner call when someone asks for an update mid-test. If you need early looks, lock a sequential rule first. Read the peeking post for the multiple-comparisons argument and the coin-flip analogy. This glossary page is only the definition.

Common mistake

A common mistake is calling an early stop "sequential" because you looked more than once. Sequential testing is a boundary or an always-valid rule chosen beforehand. Another mistake is ignoring a variant that is breaking checkout because you promised not to peek. Harm is a reason to pause. It is not a reason to declare the other version the proven winner.

How Mida works with peeking

Mida's results view can show a conversion rate, a confidence interval, a significance readout, and how many visitors are still short of the plan. That screen is a progress check. It is not an instruction to stop when the percentage looks high. The QA checklist and the significance calculator both describe peeking as declaring a winner before the planned sample. This page does not claim that Mida runs sequential testing or an automatic stopping rule. Those claims are not what the plan comparison documents.

Mida results row showing a 39.13 percent conversion rate, a confidence interval, 97.71 percent statistical significance, and about 2,800 visitors still remaining — A Mida results row from the peeking post. Visitors remaining means the planned sample is not finished.

Related A/B testing terms

FAQ

Is opening the dashboard peeking?

Opening it is not the problem. Stopping, or changing the variant, because an interim p-value or chance-to-win crossed a line you did not precommit to is peeking. A look that was written into a sequential boundary before launch is a different procedure.

How high can the false-positive rate get?

The peeking post describes repeated looks pushing the effective false-positive rate well above the stated 5%, into the range of 20 to 30 percent or higher when people stop at the first promising check. The exact number depends on how often you look and when you stop. It is not still 5%.

What should I do if a variant is clearly hurting users?

Stopping for harm is a safety decision, not a claim that the other arm won. Pause the harmful variant, write down why, and do not treat the interim gap as a finished test of the winner. The sample-size plan still applies if you restart.