A/B Testing for Product Managers

The Four Deaths

What a Test Can and Cannot Tell You

Why "Let's Just Test It" Is Often Wrong

The Shape of a Good Test

What This Course Covers

The Three Questions

When to Just Ship It

Test, or Go and Talk to Someone?

Exercise 1: Triage Your Test Ideas

Why "We Think This Will Improve Conversion" Isn't One

The Four-Part Hypothesis

Saying in Advance What Would Make You Wrong

One Change or Several?

Exercise 2: Write a Real Hypothesis

The Four Numbers

The Calculation, Worked Through

Why Most Tests Are Too Small

The Weekly Cycle Rule

Exercise 4: Do the Arithmetic Before You Build

Before You Turn It On

The A/A Test

What to Check on Day Two

The Six Things That Invalidate a Running Test

Exercise 5: Set Up Properly

What "Significant" Actually Means

Effect Size and the Range

The Flat Result

When the Result Contradicts What You Believed

Exercise 6: Read a Result Properly

Winning Isn't the Same as Worth Shipping

Rolling Out

Writing It Up

Exercise 7: Decide and Record

Peeking

Slicing Until Something Wins

Novelty and Primacy

Contamination, Mismatch and Survivorship

Exercise 8: Audit Your Own Results

Five Situations Where Testing Won't Help

What to Do With Too Little Traffic

Knowing When to Stop

Exercise 9: Diagnose Your Testing Programme

Final Project: Design, Run and Report One Real Test

Before You Build: Deciding Whether to Test

Before You Build: Writing and Stress-Testing a Hypothesis

Before You Build: Choosing Your Metrics

Before You Build: Sample Size and Duration

Running It: Pre-Launch Review

Reading It: Reading a Result Honestly

Reading It: Auditing for the Six Lies

Afterwards: The Test Readout

Checking Yourself: Is Testing Right for Your Situation?

Glossary: Words Explained

Lesson 9

Afterwards: The Test Readout

Write up my result as a one-page record. [paste hypothesis, decision table, result,

guardrails]. Structure: WHAT WE TESTED AND WHY (the change and the evidence); WHAT WE

EXPECTED (hypothesis including predicted effect size); WHAT HAPPENED (result with range,

guardrails, duration, sample, state plainly whether we detected an effect); WHAT WE

DECIDED AND WHY (referenced to the pre-declared table, noting any departure); WHAT WE

LEARNED ABOUT OUR USERS (the most valuable section, what belief has changed?); WHAT THIS

DOESN'T TELL US (honest limitations); WHAT WE'D DO DIFFERENTLY. One page. Written so

someone finding it in a year understands the reasoning, not just the number.

A/B Testing for Product Managers

Module 1Why Most A/B Tests Waste Three Weeks

Module 2Is This Even Worth Testing?

Module 3Writing a Hypothesis You Can Be Wrong About

Module 4Choosing the Metric

Module 5How Long It Needs to Run

Module 6Running It Without Ruining It

Module 7Reading the Result

Module 8Deciding and Shipping

Module 9The Six Ways Tests Lie

Module 10When A/B Testing Is the Wrong Tool

Module 11Final Project: Design, Run and Report One Real Test

Module 12Appendix: The Prompt Library

Module 13Glossary: Words Explained

Afterwards: The Test Readout