Data for AI Products
Where PMs Actually Sit in the Data Pipeline
You are not a data engineer. You are the person who defines what "correct" means, and nobody else can do that.
Let me be direct about the failure mode, because I have watched it happen in good companies with good people. The PM writes a PRD that says "the assistant should give accurate, helpful answers about billing". The ML team reads it, shrugs, and makes a hundred definitional decisions on your behalf, because they have to build something. Six weeks later you review the output and say "this is not what I meant". They are annoyed. You are annoyed. Everyone is right, and the quarter is gone.
The root cause is that "accurate" and "helpful" are not specifications. They are vibes. In a deterministic feature, vibes are survivable because the behaviour is written in code you can read. In a probabilistic feature, the behaviour is written in data, and if you do not define it, the data will define it for you.
So here is the division of labour that actually works.
| Decision | PM owns | ML / Data owns | Jointly |
|---|---|---|---|
| What behaviour are we buying with this feature | ● | ||
| What counts as a correct answer | ● | ||
| Which failures are unacceptable vs merely annoying | ● | ||
| The labeling guideline document | ● | ||
| The golden evaluation set | ● | ||
| Which data sources we are allowed to use | ● | ||
| How much data we need | ● | ||
| Architecture: prompt, RAG, or fine-tune | ● | ||
| Cleaning, splits, pipelines, infrastructure | ● | ||
| Model selection and training | ● | ||
| Latency and cost engineering | ● | ||
| What we tell users when it is wrong | ● |
Read the PM column carefully. Every single item on it is a judgement about the customer and the business. Not one of them requires you to write Python. And not one of them can be delegated to an engineer without you effectively outsourcing your product.
The two artefacts in that column that carry the most weight are the labeling guideline and the golden set. We will build both in Module 2 and Module 4. For now, understand what they are:
• The labeling guideline is the document that turns your definition of "correct" into instructions a labeler can follow without asking you a question. It is the constitution of your dataset.
• The golden set is 100 to 300 examples, written or curated by you, that encode what excellent, acceptable, and unacceptable look like for your product. It is the exam paper. If the model passes it, you ship. If your team cannot articulate what would be on that exam, you are not ready to build.
FIELD NOTE
A useful test of whether you are actually doing your job on an AI project: could you personally sit down and correctly label 50 examples right now, and would another PM on your team produce the same labels? If the answer is no, your ML team is currently guessing what you want, and they will guess in the direction that is easiest to build.
YOUR MOVE THIS WEEK
Write one page titled "what correct means for this feature". Include three examples of a great output, three of an acceptable output, and three of an output that would get you a call from your CEO. Send it to your ML lead. That page will do more for the model than three weeks of tuning.