A/B Testing for Product Managers
Before You Build: Deciding Whether to Test
Before You Build: Writing and Stress-Testing a Hypothesis
Before You Build: Choosing Your Metrics
Before You Build: Sample Size and Duration
Running It: Pre-Launch Review
Reading It: Reading a Result Honestly
Reading It: Auditing for the Six Lies
Afterwards: The Test Readout
Checking Yourself: Is Testing Right for Your Situation?
Lesson 5
Exercise 2: Write a Real Hypothesis
Take the test idea that survived Chapter 1. Write it in the four-part form.
Run the stress-testing prompt. Take question 2 seriously, if your evidence is a hunch, say so in the BECAUSE rather than dressing it up.
Write down your expected effect size as a number. If you can't justify it, look at what similar past changes achieved.
Fill in the decision table for all five outcomes, including flat and negative.
Get one colleague to read the hypothesis and the decision table and tell you what they'd argue at readout. Fix whatever they find ambiguous now.