The test is flat. Someone suggests looking at mobile users. Still flat. New users on mobile? There's something. New users on mobile in the UK who arrived from paid search, significant. Ship it.

With enough slices you will always find one that looks significant, because that's what randomness does. Twenty segments at the usual threshold means roughly one false positive by chance alone, every time.

Legitimate	Fishing
You declared the segment before the test started	You found it after the overall result disappointed
There was a reason to expect it to differ	The reason was constructed afterwards
You treat it as a hypothesis for a new test	You ship on it
You report it as exploratory	You report it as the result

Segment Findings Are Hypotheses, Not Results

It's fine to look at segments after the fact, that's how you generate ideas. What isn't fine is shipping on one. Treat it as an interesting signal, design a test for that specific segment, and see whether it replicates. About half the time it won't, which tells you what it was.

A/B Testing for Product Managers

Module 1Why Most A/B Tests Waste Three Weeks

Module 2Is This Even Worth Testing?

Module 3Writing a Hypothesis You Can Be Wrong About

Module 4Choosing the Metric

Module 5How Long It Needs to Run

Module 6Running It Without Ruining It

Module 7Reading the Result

Module 8Deciding and Shipping

Module 9The Six Ways Tests Lie

Module 10When A/B Testing Is the Wrong Tool

Module 11Final Project: Design, Run and Report One Real Test

Module 12Appendix: The Prompt Library

Module 13Glossary: Words Explained

Slicing Until Something Wins