Novelty Effect

Definition

The novelty effect is a short-lived change in behavior that happens because a variant is unfamiliar, not because it is better once people are used to it. In an A/B test it inflates early results for anything that looks new, louder, or simply different.

What Novelty Effect means in A/B testing

Visitors notice a redesigned hero, a new badge, or a moved button, and they click because it is new. That extra attention decays. The same people, on a later visit, treat the variant as the normal page. The opposite also happens: a change-aversion effect, where regulars do worse on the new page at first and catch up later. Both are time-dependent biases. A single pooled conversion rate mixes the early reaction with the later one and reports an average nobody will actually live with.

Why Novelty Effect matters

Shipping a novelty spike means the next month will not match the test. The business plans a gain that disappears, and the following experiment starts from a baseline that was never real. Novelty is also why a test that is stopped the moment it looks significant is doubly misleading: the early window is both statistically noisy and behaviorally unrepresentative. The duration calculator calls this out directly: a day-three lead can fade by day fourteen.

Example of Novelty Effect

Suppose a homepage adds a "New" badge next to the primary button. These rates are an illustration, not a Mida customer result. Week 1, the variant's click-through rate is 4.8% and the control's is 3.6%. Week 2, the variant is 4.1% and the control stays at 3.6%. Week 3, the variant is 3.7% and the control is 3.6%. The pooled three-week gap still favors the variant, because week 1 is inside the pool. The week-3 gap is 0.1 percentage points, which is not the change you would keep. A cut of first-time versus returning visitors tells the same story if only first-time visitors moved. The decision is to keep the control, or to rerun the badge only if a later week still shows a gap you would pay for.

How to use Novelty Effect

Before launch, decide the steady-state window you care about, such as "after the visitor's second session" or "days 8 through 14." Run at least that long, and run a full week so weekday and weekend traffic are both in the data. When you read the result, plot the metric by week and by new versus returning. Ship on the later window if it still clears your minimum effect. Use a holdout group after launch if the page is important enough that a fade would be expensive.

Common mistake

A common mistake is calling any disappearing lift "novelty" without looking at the mix of visitors. A promotion, a broken variant, or a sample ratio mismatch that heals itself can also shrink a gap. Check the split and the traffic sources before you blame familiarity. Another mistake is refreshing the variant's color every week to "keep novelty," which means you never measure the page you will actually leave up.

Related A/B testing terms

• Testing Period
• Confounding Variables
• Holdout Group
• Sequential Testing

FAQ

Is every early lift a novelty effect?

No. An early lift can be a real, stable change, a lucky week, or a bug that favors one arm. Novelty is the specific pattern where the gap shrinks as people get used to the new page, or where new visitors move and returning visitors do not. You need that time split or visitor split before you use the name.

How long should I wait before I blame novelty?

Long enough to include people who have seen the variant more than once, and at least one full business cycle. The duration calculator on this site warns that a day-three winner can fall back toward average by day fourteen as the sample settles and novelty fades. Two weeks is a check, not a law. A product people open daily can bore them faster than a page they see monthly.

What do I ship if week one won and week three is flat?

Do not ship on the pooled number alone. The pooled rate is mostly the excited first week. Either keep the test running until the weekly gap stabilizes, or ship only if the later, post-novelty window is still worth the change. A flat later window means the control is the better steady state.