Sequential Testing

Last updated 28 October 2026

Quick answer

Sequential testing is a way to analyze an A/B test more than once, on purpose, while keeping the false-positive rate near the level you set. It is the designed alternative to peeking.

Key takeaways

A fixed-horizon test has one look, at the sample size you set. Peeking and stopping at the first p-value under 0.05 inflates false positives.
A sequential design spends that error rate across planned looks, so an early stop needs a stricter threshold than the final look.
The method, the looks, and the rule have to be chosen before the data arrives. A boundary picked after a promising week is still peeking.

Definition

Sequential testing is a way to analyze an A/B test more than once, on purpose, while keeping the false-positive rate near the level you set. It is the designed alternative to peeking. It is not "look whenever you like and stop when the chart turns green.

What Sequential Testing means in A/B testing

A fixed-horizon test picks a sample size, an alpha such as 0.05, and a single analysis at the end. The p-value's meaning depends on that single look. If you instead peek every day and stop at the first p-value under 0.05, you have given yourself many chances to be fooled by noise, and the real Type I error is well above 5%. Sequential methods plan the looks. Group-sequential designs, such as Pocock or O'Brien-Fleming boundaries, and always-valid approaches such as a mixture sequential probability ratio test, spend alpha across those looks. Early looks demand stronger evidence. The last look sits near the original alpha. Always-valid p-values are constructed so that a crossing of 0.05 at any time still controls the error rate, which is a stronger claim than a boundary at five scheduled times. You do not get that property from an ordinary p-value.

Why Sequential Testing matters

Traffic is expensive and a bad variant does real harm, so teams want to stop early. Stopping early without a sequential rule ships noise. The QA checklist says peeking inflates false positives and that the committed rule is the pre-planned sample size unless a formal sequential method is in use. P-hacking is the broader family of choices, including this one, that make a null effect look real. Sequential testing is the version of "look early" that is still a test.

Example of Sequential Testing

You need about 20,000 visitors per variant for a fixed-horizon test at 80% power, from the sample size calculator . Plan A is one analysis at 20,000, alpha 0.05. Plan B peeks five times and stops the first time p is under 0.05. Plan B will stop some tests that Plan A would correctly call flat, because random leads happen early and then fade. Plan C schedules the same five looks in advance and uses an O'Brien-Fleming-style boundary: the first look needs a much smaller p-value than 0.05, and the last look is close to 0.05. Plan C can stop a huge early effect and still claims a 5% false-positive rate. Plan B cannot. The exact boundary values depend on the number and spacing of looks, so they are computed when you lock the design, not copied from a remembered table after you have seen the data.

How to use Sequential Testing

Choose the method, the maximum sample, and the look schedule before launch. Write the early-stop rule in the same document as the primary metric. If you do not have a boundary or an always-valid p-value, do not stop early. Use the sample size calculator for the fixed-horizon size and the duration calculator to turn that size into days. At each planned look, compare the result with that look's threshold, not with 0.05 by habit.

Common mistake

A common mistake is narrating an ordinary peek as "we used sequential testing" because you looked more than once. The name requires the error-rate math. Another is changing the look schedule after an early week looks promising. Adding a look you did not pay for in the alpha budget breaks the guarantee in the same way peeking does.

How Mida works with sequential testing

A plan comparison published on this site states that sequential testing is not a Mida feature. The decision rule the QA checklist documents is the sample size you calculated before launch. Chance to win, where a Bayesian readout is shown, is a probability on the current data, not a sequential boundary. Use the public calculators to plan the fixed horizon. Do not read this page as a claim that the Mida dashboard implements mSPRT or alpha spending.

Related A/B testing terms

FAQ

Is checking the dashboard sequential testing?

Only if the check was part of a rule you committed to, with a threshold that gets stricter because you looked. Opening the report, seeing p under 0.05, and stopping is peeking. The QA checklist says that inflates false positives and tells you to keep the pre-planned sample unless you are actually using a formal sequential method.

Why is an early look harder to pass than the last look?

The overall false-positive budget, often 5%, has to cover every look. If you give the first look the full 5%, there is nothing left for the later looks and the real error rate is higher than 5%. Early boundaries are tight so that most of the budget remains for the final, larger sample, where the estimate is more stable.

Does a Bayesian chance to win replace a sequential boundary?

Not by itself. A probability that updates every day is a different summary of the same stream of data. It still moves early, and stopping the first time it clears a round number has the same optional-stopping problem unless the stopping rule was designed for repeated looks. Treat chance to win as a description of the current data, and keep the sample plan you set.