A Test Tells You	A Test Says Nothing About
Did version B perform differently from version A?	Why it performed differently
By roughly how much, within a range	Whether you built the right thing at all
Whether that difference is likely to be real	What users were trying to do
Whether it broke anything you were watching	Anything you didn't think to measure
What happened to these users, this month	Whether the effect lasts

The second column is why testing and talking to users are complements rather than alternatives. A test tells you that the new checkout converts 6% better. It cannot tell you that it works because people finally understand the delivery timing, and without that, you don't know what to do next.

A/B Testing for Product Managers

Module 1Why Most A/B Tests Waste Three Weeks

Module 2Is This Even Worth Testing?

Module 3Writing a Hypothesis You Can Be Wrong About

Module 4Choosing the Metric

Module 5How Long It Needs to Run

Module 6Running It Without Ruining It

Module 7Reading the Result

Module 8Deciding and Shipping

Module 9The Six Ways Tests Lie

Module 10When A/B Testing Is the Wrong Tool

Module 11Final Project: Design, Run and Report One Real Test

Module 12Appendix: The Prompt Library

Module 13Glossary: Words Explained

What a Test Can and Cannot Tell You