Underpowered tests are the norm rather than the exception, and they fail in a particularly unhelpful way: they return "no significant difference" whether or not the change worked. The team concludes the change didn't help, when the honest conclusion is that the test couldn't tell.

What the Team Concludes	What Actually Happened
"The change didn't work"	The test couldn't have detected it either way
"Our users don't care about this"	We had 6,000 users and needed 60,000
"A/B testing doesn't work for us"	We've run six underpowered tests in a row
"It's directionally positive so let's ship"	That direction is indistinguishable from noise

The fix is cultural as much as technical: calculate the sample size at the idea stage, and treat an impossible sample size as a reason not to build the test. Teams that do this run fewer tests and learn more from each one.

A/B Testing for Product Managers

Module 1Why Most A/B Tests Waste Three Weeks

Module 2Is This Even Worth Testing?

Module 3Writing a Hypothesis You Can Be Wrong About

Module 4Choosing the Metric

Module 5How Long It Needs to Run

Module 6Running It Without Ruining It

Module 7Reading the Result

Module 8Deciding and Shipping

Module 9The Six Ways Tests Lie

Module 10When A/B Testing Is the Wrong Tool

Module 11Final Project: Design, Run and Report One Real Test

Module 12Appendix: The Prompt Library

Module 13Glossary: Words Explained

Why Most Tests Are Too Small