Write down, before the test starts, what result would change your mind. This takes two minutes and it's the thing that makes a readout a decision rather than a negotiation.
Result | What We'll Do | Decided In Advance |
|---|---|---|
Completion rises to 3.6% or more, guardrails fine | Ship to everyone | Yes |
Completion rises but under 3.5% | Ship anyway, the change is cheap to maintain | Yes |
Flat, no detectable difference | Don't ship. Revisit the delivery-slot problem with research. | Yes |
Completion rises but refund rate also rises | Don't ship. Investigate what we've promised. | Yes |
Completion falls | Don't ship. Our understanding of the drop-off was wrong. | Yes |
Write This Table Before You Start
Do it in the same document as the hypothesis, dated. When the result comes in and someone argues for shipping a flat result because "directionally it's positive," you point at the table. It's not that you're inflexible, it's that the decision was made when nobody had a stake in the answer, which is the only time you can make it cleanly.