Canary Testing

Definition

Canary testing is a rollout that sends a new version to a small share of users first, watches for errors and regressions, and only then expands. The name comes from the canary miners carried to detect bad air. The longer guide is What is canary testing?

What Canary Testing means in A/B testing

The new version is the canary release. The act of pushing it to a subset of production is the canary deployment. The monitoring and the go/no-go decision are the test. A common first step is 1 to 5 percent of traffic. You compare that slice with the rest of production on error rate, latency, and the behavior you said mattered. This is not an A/B test. An A/B test randomizes to estimate a metric difference. A canary limits how many people a broken release can hit.

Why Canary Testing matters

A full release makes every visitor the test. A canary makes a small group the test and leaves everyone else on the version you already trust. That only works if you can turn the new behavior off quickly. The canary post pairs the rollout with a feature flag for that reason. Automated tests still run before the canary. The canary is the production check, not a replacement for the test suite.

Example of Canary Testing

A pricing-page deploy changes how tax is calculated. You enable it for 2 percent of checkout traffic. Error rate on the canary is three times the rest of the site, and latency is up. You turn the flag off. The other 98 percent never saw the bug. You do not read the 2 percent conversion rate as a winner or a loser. The sample was there to catch the failure, not to estimate a lift.

How to use Canary Testing

Write the metrics and the fail thresholds first. Ship the code dark, then enable it for a small random slice. Watch the first window. Expand in steps only when each step is inside the limits. Roll back as soon as it is not. Use a flag so the rollback is a switch, not a deploy. The canary post walks through planning, the 1 to 5 percent start, and the difference from A/B testing.

Common mistake

A common mistake is treating a quiet canary as proof the change converts better. Quiet means you did not see the failures you listed. It says nothing about revenue unless you designed an experiment. Another mistake is picking the canary from one friendly segment, such as employees, and then treating that as production traffic.

How Mida works with canary testing

Mida's feature flags are documented for gradual exposure: turn an experience on for part of an audience, target who gets it, and connect the rollout to goals. That is the control a canary needs. The flag FAQ also says you can pause or disable a deployment when you want the change gone. This page does not describe a separate canary product. The rollout practice is the one in the canary post, using a flag as the switch.

Related A/B testing terms

• Feature Flag
• Feature Rollout
• Holdout Group
• Server-Side Testing

FAQ

How is a canary different from an A/B test?

A canary asks whether the new version is safe to keep expanding. An A/B test asks whether it beats the old version on a metric. A canary can run with no hypothesis about conversion rate. An A/B test is incomplete without one. Teams often canary a release and, separately, experiment on the part of it they want to measure.

What share of traffic should the first step use?

The canary post uses 1 to 5 percent as the typical first step, then larger steps such as 25 and 50 percent only after each step stays inside the monitoring limits. The exact first step depends on how fast you need to see errors. A tiny slice of a quiet site may not produce an error soon enough to be useful.

What do I watch before I expand?

The post names error rate, latency, and the user behavior you already care about. Set the fail line before the canary starts. If errors jump or the page slows only on the new version, stop and turn the behavior off. Do not expand because the first hour looked quiet on a handful of sessions.