What Happened	Why It Breaks the Test	What to Do
Someone changed the page mid-test	You're now testing two different things	Restart. The data before the change is unusable.
A marketing campaign sent unusual traffic	Your sample isn't your normal population	Note it. Consider excluding those days, or restart.
An outage hit one variant	That variant's numbers reflect the outage	Restart if it was more than a few hours
Another test started on the same users	The two effects are tangled	Stop one. Decide which matters more.
A public holiday or sale landed mid-test	Behaviour that week isn't typical	Extend by a week and note the anomaly
The variant was accidentally shown to everyone	No control group for that period	Restart

Restarting Is Not a Failure

The instinct is to salvage a contaminated test because three weeks feel expensive. A contaminated test doesn't produce a weaker answer, it produces an answer you can't distinguish from a wrong one, and you'll make a decision on it anyway. Restarting costs three weeks. Shipping on corrupted data can cost a quarter.

COPY THIS PROMPT - Pre-Launch Review

I'm about to launch an A/B test. Review my setup and find what

I've missed.

THE TEST: [what you're testing, both variants described]

PRIMARY METRIC: [and how it's tracked]

GUARDRAILS: [list]

DURATION: [planned weeks, and the sample size behind it]

AUDIENCE: [who's included, who's excluded]

WHAT ELSE IS HAPPENING: [other tests, campaigns, releases planned

during this window]

Please tell me:

1. Anything in my setup that could produce unusable data

2. Specific ways my tracking could be wrong in one variant but

not the other - this is the failure I most want to catch

3. Whether anything else happening in this window could

contaminate the result

4. Who I should tell about this test before it starts, and what

specifically to tell each of them

5. What I should check on day two, and what I should deliberately

not look at

A/B Testing for Product Managers

Module 1Why Most A/B Tests Waste Three Weeks

Module 2Is This Even Worth Testing?

Module 3Writing a Hypothesis You Can Be Wrong About

Module 4Choosing the Metric

Module 5How Long It Needs to Run

Module 6Running It Without Ruining It

Module 7Reading the Result

Module 8Deciding and Shipping

Module 9The Six Ways Tests Lie

Module 10When A/B Testing Is the Wrong Tool

Module 11Final Project: Design, Run and Report One Real Test

Module 12Appendix: The Prompt Library

Module 13Glossary: Words Explained

The Six Things That Invalidate a Running Test