WHAT IT MEANS
"If there were genuinely no difference between the two versions, we'd see a gap this large by pure chance less than 5% of the time."

WHAT IT DOES NOT MEAN

That the result is important
That you're 95% sure the change works
That the effect is as big as the number you measured
That a non-significant result means "no difference"

THE MOST USEFUL REFRAME
Significance answers: "is this probably real?"
Effect size answers: "is this worth anything?"

You need both. Significance alone tells you almost nothing about whether to ship.

Significant and Meaningless at the Same Time

With enough traffic, a 0.1% lift becomes statistically significant. It is real and it is worth nothing. Teams with large user bases ship these constantly, because the tool turns green and green means ship. Always ask what the effect size is in business terms before you ask whether it's significant.

A/B Testing for Product Managers

Module 1Why Most A/B Tests Waste Three Weeks

Module 2Is This Even Worth Testing?

Module 3Writing a Hypothesis You Can Be Wrong About

Module 4Choosing the Metric

Module 5How Long It Needs to Run

Module 6Running It Without Ruining It

Module 7Reading the Result

Module 8Deciding and Shipping

Module 9The Six Ways Tests Lie

Module 10When A/B Testing Is the Wrong Tool

Module 11Final Project: Design, Run and Report One Real Test

Module 12Appendix: The Prompt Library

Module 13Glossary: Words Explained

What "Significant" Actually Means