Why Most A/B Test Hypotheses Are Unfalsifiable (and How to Fix Yours)
Quick answer
If your test comes back flat, could someone point to your hypothesis and explain why it was wrong? If the honest answer is "not really, it just didn't work," the hypothesis probably wasn't specific enough to have failed in the first place. That's the real problem with most A/B test hypotheses — not that they're badly written, but that they're built in a way that no result, good or bad, could ever actually contradict them.
Key takeaways
- A falsifiable hypothesis names a mechanism and a segment, so a flat result actually tells you which one was wrong.
- The four parts worth forcing: what you observed and how, who you think it affects, what you expect to happen, and how you'll measure it.
- Before you open the test editor, try to argue against your own hypothesis in one sentence — if you can't, rewrite it.
What does "unfalsifiable" even mean for an A/B test?
A falsifiable hypothesis is one that makes a specific enough claim that a result could prove it wrong. "Changing the button color will increase conversions" doesn't clear that bar. If conversions go up, great, but you don't know if it's the color, a novelty effect, or noise. If they don't move, you've learned close to nothing — you never said why you expected the change to matter, so there's nothing to go back and check.
Compare that to: "Mobile checkout abandons at the payment step because the CTA is below the fold on smaller screens; moving it above the fold will reduce checkout abandonment for mobile sessions specifically." Now a flat result actually tells you something — either the CTA position wasn't the real friction, or mobile visitors weren't the segment you thought they were. Either way, you walk away smarter. That's the whole point of running the test.
Why vague hypotheses feel fine right up until the test ends
Nobody sets out to write a bad hypothesis. "Test X to improve Y" is fast, it fits in a ticket, and it looks like every other row in the backlog. The problem only shows up afterward, when the test is flat and the retro turns into a debate: was it the wrong page, the wrong segment, the wrong change, or just not enough traffic? Without a stated mechanism and segment up front, there's no way to tell — and that ambiguity is what quietly kills a testing program's credibility over time, one "well, that told us nothing" result at a time.
Free A/B Testing Tool
Run your next A/B test the right way
Visual editor, 15 KB script, GA4-native — and free forever up to 100,000 monthly visitors. No developer required.
The four things a real hypothesis needs
A hypothesis that can actually be proven wrong tends to have four parts: what you observed and how (an analytics number, a session recording, a support ticket — something real, not a hunch); who you think it affects (a specific segment, not "users"); what you expect to happen if you're right; and how you'll measure it. Something like: "We've observed [X] via [method]. We believe [change] will affect [segment], leading to [expected result], measured by [metric]."
That's not a coincidence of phrasing, either — it's close to the exact structure Mida's hypothesis generator forces on every input. You describe the observation in plain language, and in under 10 seconds it hands back a hypothesis in that observation → segment → outcome → metric shape. The value isn't that AI wrote it for you — it's that the structure itself won't let you skip the parts that make a hypothesis falsifiable. It's a genuinely useful gut check to run before you open the test editor, not after the results come in and you're trying to make sense of them.
A fast way to check your hypothesis before you build the test
Before setting up variants, try to argue against your own hypothesis in one sentence. If you can't — if any outcome would just get shrugged off as "well, at least we tried" — that's the tell. Rewrite it with a named segment and a stated mechanism until a flat result would actually mean something specific. It takes a couple of minutes and saves you from a retro where nobody can agree on what just happened. If you're also short on time to do that from scratch, running the observation through Mida's AI-generated hypothesis flow is a reasonable shortcut — just make sure you're still the one checking that the segment and mechanism it lands on are true to what you actually observed, not a generic best-practice guess.
The takeaway
A hypothesis doesn't need to be fancy, but it does need to be able to lose. If there's no version of the result that would surprise you or prove you wrong, you don't have a hypothesis yet — you have a guess with better formatting. Fixing that before you build the test, rather than after you're staring at a flat result, is the difference between a testing program that compounds and one that just accumulates inconclusive rows in a spreadsheet.
