A/B Testing for Product Managers
One Change or Several?
You want to change the headline, the button, and the image. Do you test all three together or one at a time? There's a real trade-off here and neither answer is always right.
Test Everything Together | Test One Thing at a Time |
|---|---|
Bigger combined effect, so a smaller sample detects it | Smaller effects, so each needs more traffic |
One test instead of three, much faster | Three tests, three times the calendar time |
You learn whether the new version is better | You learn which element did the work |
If it wins, you don't know why | If it wins, you know what to do more of |
Right when you need a decision | Right when you need understanding |
For most teams, most of the time, bundling is correct. You have limited traffic and limited weeks, and "is this new page better than the old one" is usually the question you actually need answered. Isolate single elements when you're building a durable understanding of what moves your users, and accept that it costs several times as long.
COPY THIS PROMPT - Writing and Stress-Testing a Hypothesis
Help me write a proper hypothesis for an A/B test.
WHAT I WANT TO CHANGE: [describe it]
WHAT MADE ME THINK OF THIS: [the evidence - analytics, research,
support tickets, or honestly, a hunch]
CURRENT PERFORMANCE: [the metric and its current value]
WHO I'D TEST IT ON: [which users]
Please:
1. Write it as: BECAUSE [evidence] WE BELIEVE [change] WILL CAUSE
[metric to move by amount] FOR [users].
2. Challenge my evidence. Is the BECAUSE actually supported, or
have I dressed up a preference as a finding? Be blunt.
3. Is my expected effect size realistic? Tell me if it looks
optimistic given what typically happens with changes like this.
4. Draft the decision table: for each plausible outcome, what
should we do? Include the flat result and the negative one.
5. Name one alternative explanation for my evidence that I might
have missed - a different reason the drop-off could be happening.