Almost every product has a weekly rhythm. Weekend users differ from weekday users. Monday differs from Friday. If your test runs for ten days, it contains one and a half weeks and is weighted toward whichever days appeared twice.
Rule | Why |
|---|---|
Always run in whole weeks | Otherwise your sample over-represents some days |
Minimum one full week, even with huge traffic | You need one complete cycle to see weekday and weekend behaviour |
Never stop on a Tuesday because it looks good | That's peeking (Chapter 8) and it's the most common error there is |
Avoid starting or ending on a holiday | Unusual traffic on either end distorts the comparison |
Beware seasonal periods | A test run across a sale period measures the sale as much as your change |
COPY THIS PROMPT - Working Out Your Sample Size and Duration
Help me work out how long my A/B test needs to run, and whether
it's feasible at all.
MY NUMBERS
Current rate of the primary metric: [e.g. 3.2% convert]
Users reaching this point per week: [number]
Share I can include in the test: [usually 100%, sometimes less]
Smallest lift worth detecting: [e.g. +10% relative]
Please:
1. Estimate the sample size needed per variant at 95% confidence
and 80% power. Show your working in plain language.
2. Convert that to a duration in whole weeks given my traffic.
3. Show me how that duration changes if the true effect were
half the size I'm hoping for, and double the size.
4. Tell me plainly whether this test is feasible. If it needs more
than about 6 weeks, say so and give me my four options: test
somewhere with more traffic, make a bolder change, accept lower
confidence, or don't test.
5. Flag anything about my situation that would make these estimates
unreliable - low baseline, high variance, seasonal traffic.
Note that this is an estimate for planning, not a formal power
calculation, and tell me when I'd need someone to do it properly.