Data for AI Products
You Already Know Analytics Data. Training Data Is a Different Beast.
Analytics data is evidence you read. Training data is behaviour you install.
Let us start with something you have done a hundred times. You open Mixpanel, you look at a drop-off between step two and step three of onboarding, you form a hypothesis, you run a test, you ship. In that loop, the data is evidence. It sits still. You look at it. You decide. If the data is a bit messy, you squint, you apply judgement, and you move on. Nobody dies.
Now change one thing. Instead of you reading the data, a model reads it and copies it. Every pattern in that data, including the ones you did not notice and would never have endorsed, becomes part of how the product behaves in front of a customer. The data is no longer evidence. It is instruction.
That single shift is the reason training data is a new skill even for very senior PMs. In analytics, an error is a wrong conclusion that you can correct next week. In training data, an error is a wrong behaviour that is now shipped, silent, and hard to trace back to its cause.
Fig 1.1 - The same word, two jobs. In the top loop a human is the interpreter. In the bottom loop the data is the interpreter.
Here is the practical breakdown. Keep this table somewhere you can find it, because it is the mental model that stops most PM mistakes in month one of an AI project.
| Dimension | Product / analytics data | Training data |
|---|---|---|
| Purpose | Help a human decide | Teach a machine to act |
| Consumer | You, your exec, your dashboard | A model, an eval harness, a pipeline |
| Unit | Event, session, user, cohort | Example, label, pair, document chunk |
| Tolerance for noise | Fairly high. You eyeball and adjust. | Low. Noise becomes learned behaviour. |
| What "more" gives you | Statistical confidence | Coverage of new situations, only if the new data is different |
| Bias risk | You draw a skewed conclusion | The product treats a group of users worse, at scale, every day |
| Failure signature | A wrong chart | A confident wrong answer with no chart at all |
| Who owns quality | Data analytics team | Nobody, unless the PM assigns it |
Look at that last row again. It is the row that hurts. Analytics quality has an owner in almost every company. Training data quality usually does not, because it sits between the ML engineer who assumes the data is someone else's problem, the ops team who created it for a completely different reason, and the legal team who was never told it would be used this way.
That gap is your job.
WORKED EXAMPLE
A fintech wants an AI assistant that answers "why was my transaction declined". Analytics view: 4.2% of transactions decline, top three reasons are insufficient balance, risk flag, and bank downtime. Useful for a roadmap. Useless as training data. The training view asks completely different questions. Do we have real customer questions paired with the correct explanation? Who wrote those explanations? Were they accurate? Did the agent who wrote them have access to the risk reason, or did they guess? If agents guessed, the model will learn to guess with confidence, which is the worst possible outcome for a money product.
Notice that none of those questions are technical. They are product questions about provenance, intent, and truth. You do not need to know how a transformer works to ask them. You do need to know that they must be asked before anyone writes code, because by the time the model is trained, the wrong answer is baked in.
YOUR MOVE THIS WEEK
Take one AI feature on your roadmap. Write two lists side by side. Left column, the data you would use to report on that feature. Right column, the data you would use to teach that feature. If the right column is empty or full of guesses, you have found the real project, and it is not a model project.