A/B Testing for Product Managers
The Calculation, Worked Through
THE MEAL-KIT SIGNUP TEST
Baseline conversion | 3.2% |
|---|---|
Effect we want to detect | +12% relative (to 3.6%) |
Confidence | 95% |
Power | 80% |
-> Sample needed: roughly 31,000 per variant, roughly 62,000 total
Weekly traffic to this page: 40,000. Share included in the test: 100%.
62,000 / 40,000 = 1.55 weeks. Round up to whole weeks -> 2 WEEKS
Now the same test, hoping for a +4% lift instead:
-> Sample needed: roughly 280,000 per variant, 14 WEEKS
Same change. Same traffic. A quarter of the year, because of one number in the hypothesis.
What people assume | 2 weeks (detecting a 12% lift vs a 4% lift, identical traffic) |
|---|---|
What's actually true | 14 weeks (same traffic, same test) |
What to Do When the Answer Is 14 Weeks
You have four honest options and "run it for three weeks anyway" is not one of them. Test somewhere with more traffic. Make a bolder change with a larger expected effect. Accept a lower confidence level and say so in the readout. Or don't test it, ship and watch. Picking one of those four deliberately is a much better position than running an underpowered test and reading tea leaves.