Five Situations Where Testing Won't Help

You don't have enough traffic. If your realistic sample size needs six months, testing is not available to you for this change. This is the most common situation for B2B products and early-stage companies, and pretending otherwise produces a long series of uninterpretable flat results. Use research, ship-and-watch, or staged rollouts instead.

The change is too big to isolate. A full redesign changes a hundred things at once. You can test whether the new version beats the old one, which is a real question, but you'll learn nothing about why, and if it loses you won't know which of the hundred things caused it. For changes this size, research before and careful monitoring after are worth more than the comparison.

The effect takes months to appear. Changes to onboarding that affect annual retention, pricing changes that affect lifetime value, anything where the outcome arrives long after the change. You can test a proxy, but only if you've verified the proxy predicts the outcome, and if you haven't, you're optimising something you haven't connected to anything that matters.

You'd ship it regardless. Compliance, accessibility, security, fixing something broken, or anything the company has already committed to. Running a test here is theatre, and it costs you traffic that another test could have used.

You need to know why, not whether. "Should we redesign the dashboard?" is not a testable question until you know what people are failing to do on it. A test compares two guesses; research generates better guesses. If your last three tests were flat, this is probably your situation.