Signal | What It Means |
|---|---|
Three flat tests in a row on the same surface | You're testing variations on a guess. Go and do research. |
Every test needs more than 8 weeks | Your traffic doesn't support this programme. Change the method. |
You're arguing about the result more than acting on it | The decision table was missing or ignored |
You've shipped ten wins and the top-line metric hasn't moved | Your primary metrics aren't connected to the outcome |
The test would take longer than the decision can wait | Decide without it and say that's what you're doing |
The fourth row is worth sitting with. A programme that produces a steady stream of wins while the business metric stays flat is measuring the wrong things, and no amount of additional testing rigour will fix that. It's a Chapter 3 problem wearing a Chapter 6 costume.
COPY THIS PROMPT - Checking Whether Testing Is Right for You
Help me work out whether A/B testing is the right method for my
situation, or whether I should be doing something else.
MY PRODUCT: [what it is, who uses it]
TRAFFIC: [users per week reaching the surfaces I'd test]
TYPICAL BASELINE RATES: [conversion or completion rates]
MY LAST FIVE TESTS: [what each tested, how long it ran, and
what the result was - include the flat ones]
WHAT I WANT TO TEST NEXT: [description]
Please tell me directly:
1. Given my traffic and baselines, what size of effect can I
realistically detect in a reasonable timeframe?
2. Looking at my last five tests - is there a pattern suggesting
I'm testing variations on a guess rather than testing a
well-understood problem?
3. For the thing I want to test next: test it, ship and watch,
staged rollout, or research first? Pick one.
4. If A/B testing isn't well suited to my situation generally,
say so plainly and tell me what a better approach would be
for a product at my scale.
Don't be encouraging. I'd rather know.