A/B Testing for Product Managers
What to Do With Too Little Traffic
Most teams don't have enough traffic to test most things, and this is rarely said out loud. The alternatives are legitimate methods, not consolation prizes.
Instead of a Test | How It Works | What You Give Up |
|---|---|---|
Watch five users | Structured observation of people doing the task | Quantification, you learn why, not how much |
Ship and watch | Release to everyone, monitor the metric for a fortnight | Can't separate your change from everything else |
Staged rollout | Release gradually, watching for problems at each stage | A proper comparison, but you catch disasters |
Before and after | Compare the period before to the period after | Seasonality and other changes confound it |
Test a bolder change | Make a bigger change so the effect is detectable | Precision about what specifically worked |
Test on the highest-traffic surface | Move the test to where the volume is | You're testing a different thing |
Bolder Changes Are Underrated
If your traffic only supports detecting a 15% lift, then small refinements are untestable and you should stop proposing them. That constraint pushes you toward bigger, more distinctive changes, which are usually more interesting anyway, and which produce clearer learning whichever way they go.