A Great Place to Upskill
Get the latest updates from Product Space
© ACADENCE EDUTECH PRIVATE LIMITED
A Great Place to Upskill
Get the latest updates from Product Space
How to know whether your change actually worked, and tell a real result from noise
Learn to design, run, and honestly read A/B tests: hypotheses, sample size, guardrail metrics, and the six ways tests quietly lie.
The four ways a test fails before it starts
What a test can and cannot tell you
Why "we'll just test it" is often the wrong answer
What this course covers and what it doesn't
Four ways a test is doomed before it starts, what a test can actually tell you, and why "let's just test it" is often the wrong instinct.
The three questions that kill half your test ideas
When to just ship it
When to test, and when to go and talk to someone instead
The cost of a test, honestly counted
Half of all test ideas shouldn't be tested. This chapter is about finding out which half, before you build anything.
Why "we think this will improve conversion" isn't a hypothesis
The four-part hypothesis
Saying in advance what would make you wrong
One change or several: the honest trade-off
Why a hypothesis needs to be specific enough to fail, and how to write one that produces a real decision instead of a debate.
The primary metric, and why there's exactly one
Guardrail metrics: what you must not break to win
Metrics that can be won by making the product worse
Leading and lagging: what you can actually measure in three weeks
One primary metric, a few guardrails, and an honest look at whether your metric can be won by making the product worse.
Sample size without the statistics degree
The four numbers that decide everything
Why most tests are too small to detect what you're hoping for
The weekly cycle rule, and why you never stop on a Tuesday
The arithmetic that decides whether your test is capable of answering your question, explained without a statistics degree.
The setup checklist before you turn it on
The A/A test, and why it's worth the week
What to check on day two
The six things that invalidate a test mid-flight
The setup checks that prevent a wasted three weeks, and the six things that invalidate a test mid-flight.
What "significant" actually means, in plain words
The flat result: the most common and most misread outcome
Effect size: the number that matters more than the p-value
When the result contradicts what you believed
What the numbers actually mean, how to handle the flat result, and what to do when the answer isn't the one you wanted.
Winning isn't the same as worth shipping
The decision the test cannot make for you
Rolling out, and watching for the effect to fade
Writing it up so the next person benefits
Winning isn't the same as worth shipping. How to decide, roll out safely, and write up results so the next person benefits.
Peeking, and why it's the most common error
Slicing until something is significant
Novelty and primacy effects
Contamination, sample mismatch, and survivorship
Each of these produces a confident, wrong answer, and each one is committed by good teams with good intentions.
Five situations where testing won't help
Too little traffic: what to do instead
Changes too big to isolate
Knowing when to decide without a test
The most valuable judgement in this course: knowing when to test, and when to do something else entirely.
Part 1: A hypothesis, metrics and a sample size calculation
Part 2: A run, with a mid-flight check and no peeking
Part 3: A readout with a decision and an honest limits section
One properly designed, properly run, honestly reported A/B test, applying everything from the course to a real test on a real product.
Before you build: deciding whether to test, writing a hypothesis, choosing metrics, sample size and duration
Running it: pre-launch review
Reading it: reading a result honestly, auditing for the six lies
Afterwards: the test readout
Checking yourself: is testing right for your situation
Every copy-and-use prompt from the course, organized by the stage of testing it supports.
Plain-language definitions of every term used in the course, from A/A test to underpowered, for quick reference while working.
Akhil Yash Tiwari
Founder
There's no fixed number, it depends on your baseline rate, traffic, and the smallest effect worth detecting. As a rule, always run in whole weeks (minimum one full cycle) and calculate sample size before you build anything. A test needing 14 weeks isn't unusual if you're hoping to detect a small lift
They're often confused but mean different things. "Not significant" just means the test couldn't confirm a difference, it doesn't mean there is no difference. A flat result could mean (a) there's genuinely no effect, (b) your test was too small to detect the effect, or (c) the effect exists only for some users. Check whether your confidence range is narrow-and-centered-on-zero versus wide-and-spanning-both-directions to tell which.
That's "peeking," the single most common error in A/B testing. Significance fluctuates naturally during a run, so checking repeatedly gives you multiple chances to catch a false positive. Decide your end date in advance and don't look at outcomes until then.
Exactly one primary metric, declared in writing before you start. Every additional metric you check increases your odds of a false positive purely by chance. Use 2-3 guardrail metrics to catch ways the primary metric could be "won" by making the product worse (e.g., signups up but refunds also up).
Not necessarily. Significance tells you the result is probably real, not that it's worth anything. A 0.1% lift can be statistically significant and still meaningless in business terms. Always check the effect size and range, not just the significance flag, and weigh in maintenance cost, guardrails, and risk before shipping.
Yes, and for most teams most of the time, bundling is the right call. You'll learn whether the new version is better with less traffic and time, but not why. Isolate single elements only when you're building a durable understanding of what moves users, and accept it takes several times longer.
You have legitimate alternatives: watch five users directly, ship-and-watch, staged rollouts, or test a bolder change (bigger effects are easier to detect with less traffic). Running an underpowered test anyway usually just produces a false "it didn't work" conclusion.
Six main things: someone changes the page mid-test, an unusual marketing campaign hits, an outage affects one variant, another test overlaps the same users, a holiday/sale lands mid-test, or a variant accidentally gets shown to everyone. Most of these mean restarting rather than trying to salvage the data.
This is actually the most valuable kind of result. Don't slice the data looking for a segment where it "won," don't quietly bury it, and don't ship anyway based on qualitative feedback. Report the result as designed, then ask what you got wrong in your original reasoning, usually the evidence behind the hypothesis was thinner than it felt.
Five situations: you don't have enough traffic to detect a meaningful effect, the change is too big to isolate (like a full redesign), the effect takes months to show up, you'd ship the change regardless of the result (compliance, bug fixes), or you need to understand why users behave a certain way rather than whether one option beats another. In those cases, research, ship-and-watch, or staged rollouts serve you better.
