Checking Yourself: Is Testing Right for Your Situation? | A/B Testing for Product Managers

A/B Testing for Product Managers

The Four Deaths

What a Test Can and Cannot Tell You

Why "Let's Just Test It" Is Often Wrong

The Shape of a Good Test

What This Course Covers

The Three Questions

When to Just Ship It

Test, or Go and Talk to Someone?

Exercise 1: Triage Your Test Ideas

Why "We Think This Will Improve Conversion" Isn't One

The Four-Part Hypothesis

Saying in Advance What Would Make You Wrong

One Change or Several?

Exercise 2: Write a Real Hypothesis

The Four Numbers

The Calculation, Worked Through

Why Most Tests Are Too Small

The Weekly Cycle Rule

Exercise 4: Do the Arithmetic Before You Build

Before You Turn It On

What to Check on Day Two

The Six Things That Invalidate a Running Test

Exercise 5: Set Up Properly

What "Significant" Actually Means

Effect Size and the Range

The Flat Result

When the Result Contradicts What You Believed

Exercise 6: Read a Result Properly

Winning Isn't the Same as Worth Shipping

Exercise 7: Decide and Record

Slicing Until Something Wins

Novelty and Primacy

Contamination, Mismatch and Survivorship

Exercise 8: Audit Your Own Results

Five Situations Where Testing Won't Help

What to Do With Too Little Traffic

Knowing When to Stop

Exercise 9: Diagnose Your Testing Programme

Final Project: Design, Run and Report One Real Test

Before You Build: Deciding Whether to Test

Before You Build: Writing and Stress-Testing a Hypothesis

Before You Build: Choosing Your Metrics

Before You Build: Sample Size and Duration

Running It: Pre-Launch Review

Reading It: Reading a Result Honestly

Reading It: Auditing for the Six Lies

Afterwards: The Test Readout

Checking Yourself: Is Testing Right for Your Situation?

Glossary: Words Explained

Lesson 10

Checking Yourself: Is Testing Right for Your Situation?

Help me work out whether A/B testing suits my situation. MY PRODUCT: [what, who uses it].

TRAFFIC: [users per week on testable surfaces]. TYPICAL BASELINES: [rates]. MY LAST FIVE

TESTS: [what each tested, duration, result, include the flat ones]. WHAT I WANT TO TEST

NEXT: [description]. Tell me directly: what effect size can I realistically detect in a

reasonable timeframe? Looking at my last five, is there a pattern suggesting I'm testing

variations on a guess rather than a well-understood problem? For the next thing: test,

ship and watch, staged rollout, or research first? And if testing isn't well suited to my

scale generally, say so plainly and tell me what is. Don't be encouraging.