A/B Testing for Product Managers
Before You Build: Deciding Whether to Test
Before You Build: Writing and Stress-Testing a Hypothesis
Before You Build: Choosing Your Metrics
Before You Build: Sample Size and Duration
Running It: Pre-Launch Review
Reading It: Reading a Result Honestly
Reading It: Auditing for the Six Lies
Afterwards: The Test Readout
Checking Yourself: Is Testing Right for Your Situation?
Lesson 5
Exercise 4: Do the Arithmetic Before You Build
Take your hypothesis from Chapter 2. Pull out the baseline rate, weekly traffic, and your expected effect size.
Run the sample size prompt. Write down the duration in whole weeks.
Now redo it assuming the true effect is half what you hope. That's the more realistic case, write down that duration too.
If the realistic duration is over six weeks, choose one of the four options explicitly. Write down which and why.
Put the planned end date in the calendar now, before the test starts. That date is your defence against peeking.