Data for AI Products
The Data Lifecycle, Explained for Non-Technical PMs
Seven stages. You do not need to run any of them. You need to know which one is currently on fire.
Ask an ML engineer to explain the data pipeline and you will get a whiteboard full of arrows, a mention of Airflow, and a strong feeling that you should not have asked. So here is the version that matters to you, stripped down to seven stages, with the one question a PM should answer at each.
Fig 1.3 - Seven stages. Note the loop at the bottom. Once you ship, your product starts generating the data that trains its next version, for better or worse.
| Stage | What actually happens | The PM question that saves the project |
|---|---|---|
| 01 Source | Data is pulled from product logs, CRM, docs, vendors, or public sources | Do we have the legal right to use this for training, specifically? Not "do we own it". "Are we allowed to train on it?" |
| 02 Clean | Duplicates removed, junk stripped, PII redacted, formats normalised | What did we throw away, and would any of it have represented a real user? |
| 03 Label | Humans (or a bigger model) attach the correct answer to each example | Who is qualified to say what the right answer is here, and do they agree with each other? |
| 04 Split | Data is divided into training, validation, and test sets | Is the test set genuinely held out, and does it look like next quarter's traffic or last year's? |
| 05 Train | The model learns, or is given retrieval access, or is prompted | Nothing. Let them work. This is the stage where PMs add the least value. |
| 06 Evaluate | Model is scored on the held-out set and against a golden set | Does this metric correspond to something a customer would notice? If accuracy improves 4% and no user can tell, we have not shipped value. |
| 07 Monitor | Live inputs and outputs are logged, drift and failures are tracked | How will we find out we are wrong before the customer tells us on Twitter? |
Two things about that table deserve emphasis.
First, stage 05 is the only one where the honest answer is "stay out of the way". That is a feature, not an insult. Most of the value a PM adds to an AI project is created before training starts and after evaluation ends. If you spend your energy trying to have opinions about learning rates, you are contributing nothing and neglecting the two stages nobody else is looking after.
Second, stage 04 is where careers quietly go to die. A "leaky" split, where information from the test set has crept into the training set, produces beautiful offline numbers and a product that collapses in production. The classic version in consumer products: you split by row instead of by user, so the same customer's data appears in both sets, and the model has effectively seen the answers before the exam. Ask about this. The question is short: "how did we split, by row or by entity, and by time?"
VOCABULARY YOU NOW OWN
Training set, the examples the model learns from. Validation set, used to tune choices during development. Test set, touched only at the end, to get an honest score. Golden set, a small, hand-curated set of cases that represent what "good" means to your business, often written by the PM. Drift, when the world changes and yesterday's data stops describing today's users.
YOUR MOVE THIS WEEK
Draw the seven boxes for your own feature on a page and write a name next to each one. If any box has no name, that is your risk register for the quarter. In most companies, boxes 02, 03, and 07 have no name.