A/B Testing for Product Managers
Before You Build: Deciding Whether to Test
Before You Build: Writing and Stress-Testing a Hypothesis
Before You Build: Choosing Your Metrics
Before You Build: Sample Size and Duration
Running It: Pre-Launch Review
Reading It: Reading a Result Honestly
Reading It: Auditing for the Six Lies
Afterwards: The Test Readout
Checking Yourself: Is Testing Right for Your Situation?
Lesson 4
Exercise 1: Triage Your Test Ideas
List the next five things your team is considering testing.
For each, answer questions 1 and 2 from section 1.1 in writing. Be honest, "we'd ship it anyway" is a common and useful answer.
Run the decision prompt on the two that survive. Pay attention to question 2's arithmetic.
Sort all five into: test it, ship and watch, or research first. Write one line of justification each.
Count how many of the five you were going to test and now won't. For most teams it's two or three, and that's weeks of time recovered.