Your redesign, which everyone liked, converts 8% worse. This is uncomfortable and it's also the most valuable kind of result, because it's the one you couldn't have reached by reasoning.
What Teams Do | What to Do Instead |
|---|---|
Look for a segment where it won | Report the result as designed, then investigate separately |
Question the tracking (only now) | Verify tracking, but you should have done this at setup |
Run it again hoping for a different answer | Ask what the result is telling you about your model of users |
Ship it anyway because of "qualitative feedback" | Reconcile the two honestly, both are evidence |
Quietly not mention it | Write it up. Negative results save the next person three weeks. |
The most useful question after a losing test is not "what went wrong with the test" but "what did I believe about users that turns out to be wrong?" That's where the value is. The BECAUSE clause in your hypothesis is the thing to re-examine, usually the evidence was thinner than it felt at the time.
COPY THIS PROMPT - Reading a Result Honestly
Help me interpret an A/B test result without fooling myself.
MY HYPOTHESIS: [the four-part hypothesis you wrote]
MY DECISION TABLE: [what you said in advance you'd do for each
outcome]
THE RESULT
Primary metric - control: [value] variant: [value]
Range or confidence interval: [if you have it]
Significance: [what the tool says]
Sample size reached: [actual vs planned]
Guardrails: [each one, control vs variant]
Duration: [actual weeks run]
Please:
1. State plainly what this result does and does not tell me.
If it's flat, tell me which of the three kinds of flat it is.
2. Is the effect size meaningful in business terms, separately
from whether it's significant?
3. Does my decision table give a clear answer here? If I'm now
tempted to depart from it, name that temptation explicitly.
4. What are the honest limitations of this result - what can't
I conclude from it?
5. What's the most likely thing I got wrong in my original
reasoning, given this outcome?
Be direct. I'd rather be corrected now than in the readout.