Problem	What It Is	How It Misleads You
Contamination	Users see both variants, shared devices, logged out, word of mouth	Blurs the difference; real effects look smaller
Sample mismatch	The split isn't 50/50, or the two groups differ systematically	Any difference might be the groups, not the change
Survivorship	You only measure users who reached a later step	A change that filters people out looks like it improved quality
Interaction	Another test is running on the same users	The two effects are tangled and neither is readable

Survivorship Is the Subtle One

Imagine a change that makes signup harder. Fewer people finish, but the ones who do are more committed, so first-order rate among completers goes up. Measured at the wrong point, the change looks like an improvement while actually costing you customers. Always measure from the top of the funnel, not from the step after your change.

COPY THIS PROMPT - Auditing a Result for the Six Lies

Before I act on this A/B test result, check it for the common ways

tests mislead people.

THE TEST: [what you tested]

PLANNED DURATION: [weeks] ACTUAL DURATION: [weeks]

WHEN I LOOKED AT RESULTS: [be honest - did you check early?]

THE RESULT: [overall figures]

SEGMENTS I LOOKED AT: [list all of them, including ones you

checked and discarded]

WHICH SEGMENTS WERE DECLARED IN ADVANCE: [list]

SPLIT ACHIEVED: [actual percentages per variant]

OTHER TESTS RUNNING: [any]

WHERE I MEASURED FROM: [which step in the funnel]

Check for:

1. Peeking - did I stop early or check repeatedly?

2. Segment fishing - are any of my findings post-hoc?

3. Novelty or primacy - is the effect likely to be temporary?

4. Contamination - could users have seen both versions?

5. Sample mismatch - is the split and the population clean?

6. Survivorship - am I measuring from the right starting point?

For each, say whether it's a concern here and what it would mean

for my conclusion. Then tell me how much I should trust this

result overall. Be blunt.

A/B Testing for Product Managers

Module 1Why Most A/B Tests Waste Three Weeks

Module 2Is This Even Worth Testing?

Module 3Writing a Hypothesis You Can Be Wrong About

Module 4Choosing the Metric

Module 5How Long It Needs to Run

Module 6Running It Without Ruining It

Module 7Reading the Result

Module 8Deciding and Shipping

Module 9The Six Ways Tests Lie

Module 10When A/B Testing Is the Wrong Tool

Module 11Final Project: Design, Run and Report One Real Test

Module 12Appendix: The Prompt Library

Module 13Glossary: Words Explained

Contamination, Mismatch and Survivorship