A/B Testing for Product Managers
The Four-Part Hypothesis
THE SHAPE
BECAUSE [what you observed, the evidence]
WE BELIEVE [changing X to Y]
WILL CAUSE [this specific metric to move by roughly this much]
FOR [this specific group of users]
WORKED EXAMPLE
BECAUSE 61% of people who abandon the signup flow do so on the delivery-slot step, and 9 of 12 user sessions showed confusion about whether slots were guaranteed,
WE BELIEVE showing a confirmed delivery window with a named date instead of a generic "choose a slot" picker,
WILL CAUSE signup completion to rise from 3.2% to at least 3.6% (a relative lift of 12%),
FOR first-time visitors on mobile.
Each part does a job. The BECAUSE forces you to have evidence, which kills ideas that are really just preferences. The WILL CAUSE forces a number, which determines your sample size and prevents post-hoc reinterpretation. The FOR forces you to name your population before you see the data, which is what stops you slicing your way to a result in Chapter 8.
The Number in "WILL CAUSE" Is Not Optional
Without an expected effect size you cannot calculate a sample size, which means you cannot know how long to run, which means you'll stop when the result looks good. That one omission causes more bad decisions than any other single mistake in A/B testing.