Before You Turn It On

Ten minutes of checking here saves the most painful outcome in testing: running for three weeks and then discovering the data is unusable.

THE PRE-LAUNCH CHECKLIST

THE PLAN

• ☐ Hypothesis written, with an expected effect size

• ☐ Primary metric declared in writing, dated

• ☐ Guardrail metrics declared

• ☐ Decision table filled in for every outcome

• ☐ Planned end date in the calendar

THE SETUP

• ☐ Both variants actually work, someone has used each

• ☐ The split is random, and 50/50 unless you chose otherwise

• ☐ A user gets the same variant every visit

• ☐ The metric fires correctly in both variants

• ☐ Nothing else is being changed on this surface

• ☐ Other running tests don't overlap this audience

THE PEOPLE

• ☐ Marketing knows not to run a campaign here

• ☐ Support knows a test is running

• ☐ Whoever will ask for early results knows they can't have them

The Metric Firing in Both Variants

This is the one that catches people. The tracking was built for the original version and quietly doesn't fire on the new one, so variant B looks catastrophically worse. Teams have killed good changes this way. Have someone complete the journey in both variants and confirm the event appears, before you launch.