A/B Testing for Product Managers
Before You Build: Deciding Whether to Test
Before You Build: Writing and Stress-Testing a Hypothesis
Before You Build: Choosing Your Metrics
Before You Build: Sample Size and Duration
Running It: Pre-Launch Review
Reading It: Reading a Result Honestly
Reading It: Auditing for the Six Lies
Afterwards: The Test Readout
Checking Yourself: Is Testing Right for Your Situation?
Lesson 3
What to Check on Day Two
You check the setup, not the results. This distinction matters, looking at the outcome on day two is peeking, and it's the subject of Chapter 8.
Check This on Day Two | Do Not Look at This |
|---|---|
Are both variants receiving traffic? | Which variant is winning |
Is the split roughly 50/50? | The conversion rates |
Are metrics firing in both variants? | Statistical significance |
Any spike in errors or complaints? | Whether you can stop early |
Does variant B look right on mobile? | Anything that would tempt you to stop |
A sensible practice: have someone who isn't the test owner do the day-two check, and report only on the left-hand column. It removes the temptation entirely rather than relying on your self-discipline at a moment when you're curious.