A/B Testing for Product Managers
Rolling Out
You've decided to ship. A staged rollout with a few checks catches the problems a test wouldn't have shown you.
A STAGED ROLLOUT
Day 0 | Ship to 25% of users. Watch: errors, support contacts, the guardrails. |
|---|---|
Day 3 | If nothing has broken, go to 50%. Check the primary metric is behaving as the test predicted. If it isn't, that's worth pausing for. |
Day 7 | Go to 100%. Remove the old code path once you're confident. |
Week 4 | Check the effect is still there. A gain that has faded was probably novelty. |
Week 12 | Check the lagging metric your primary was a proxy for. This is the step everyone skips and it's the one that tells you whether the proxy was ever worth trusting. |
The Week-Twelve Check Is the Valuable One
If your primary metric was a proxy for something longer-term, this is where you find out whether the proxy held. Over several tests it also tells you whether your entire measurement approach is sound. Almost nobody does it, which is why teams can optimise a proxy for two years before discovering it was only loosely connected to the outcome.