Data for AI Products
A PM's Practical Overview of Data Labeling
Labeling is not data entry. It is your product policy, written one example at a time.
Labeling means attaching the correct answer to a piece of data so a model can learn from it. That sounds mechanical. It is not. The moment you write the guideline that tells a labeler what "correct" is, you are writing product policy, and every ambiguity in that document becomes a defect in your model.
Start with the types, because PMs often assume "labeling" means one thing when it means five.
| Type | What the human does | Typical use | Relative cost |
|---|---|---|---|
| Classification | Picks a category from a list | Intent detection, ticket routing, moderation | Low |
| Extraction / span tagging | Highlights the relevant part of a document | Invoice parsing, KYC, entity extraction | Medium |
| Demonstration writing | Writes the ideal answer from scratch | Fine-tuning an assistant, tone and format | High |
| Preference ranking (RLHF style) | Sees two model outputs, picks the better one and says why | Aligning an assistant to your taste | High |
| Critique / rubric scoring | Scores an output against a rubric, usually 1 to 5, per dimension | Evaluation sets, quality monitoring | High, but small volume |
Notice how the cost climbs as the task gets more subjective. That is not a coincidence, it is the central economics of labeling. Anyone can tell you whether a photo contains a car. Very few people can tell you whether a financial advisory answer is compliant, well-reasoned, and appropriately hedged.
Who does the labeling, and what it costs
You have four realistic options, and the correct answer is usually a mix of them rather than one.
| Approach | When it works | Indicative economics |
|---|---|---|
| Crowd platforms (Mechanical Turk, Appen) | High volume, objectively verifiable answers, low stakes | Roughly $4 to $12 per annotator hour depending on region and skill, per 2025 to 2026 vendor benchmarks. Needs redundancy and aggressive QA. |
| Managed vendors (iMerit, Sama, CloudFactory, Labelbox and similar) | Moderate volume, some judgement required, you need an audit trail | Higher unit price, but they absorb recruitment, training, and QA overhead. Multi-layer QA typically adds 30% to 60% to base annotation time. |
| Domain experts (doctors, lawyers, your own senior agents) | The answer requires real expertise and being wrong is expensive | Expert annotation commonly adds 3x to 10x on top of base cost. Scale AI has reportedly paid $30 to $60 per hour for PhD-level annotators. |
| Model-assisted labeling (an LLM pre-labels, humans review) | Almost always worth trying first, as a throughput multiplier not a replacement | Vendors report large throughput gains with human-in-the-loop review. The saving is real, the risk is that the model's blind spots become the dataset's blind spots. |
The market context is worth one sentence, because it explains why your ML lead's face changes when you say "let us just label more data". Analysts size the data labeling market somewhere in the region of $20 billion in 2024 with sustained growth above 20% a year, and Meta's reported $15 billion investment in Scale AI in mid-2025 tells you what the industry now thinks labeled data is worth. This is not a rounding error in your budget. Treat it as a line item.
Fig 2.1 - The mistake is not choosing the expensive option. The mistake is choosing one option for everything.
The guideline document is the actual deliverable
Here is a real pattern I want you to internalise. Two labelers get the same customer message: "I want to cancel." One tags it as churn_intent. The other tags it as subscription_cancel_request. Neither is wrong. Both are following common sense. And your model is now learning that two identical messages have two different correct answers, which teaches it precisely nothing.
A good labeling guideline prevents that. It contains:
• A one-line definition of every label, written so a new joiner cannot misread it
• Two or three positive examples per label, taken from real data, not invented
• Two or three near-miss examples per label, with an explanation of why they belong somewhere else. This is the highest-leverage section and the one everybody skips.
• An explicit rule for ambiguity. Usually: "if you are unsure between two labels, apply X and flag it." Never leave labels to invent their own tie-breaker.
• A rule for what to do with data that should not be there at all, such as PII or abuse
Write version one yourself. Then have three people label the same 50 examples independently, and measure how often they agree. That agreement number is the ceiling on your model's performance, because a model cannot be more consistent than the data it learned from. If your humans agree 70% of the time, do not expect the model to hit 95%.
YOUR MOVE THIS WEEK
Label 50 examples yourself before you commission anyone else to label 5,000. You will discover the ambiguities in your own definitions within the first ten, and each one you fix now costs nothing. Each one you discover after the vendor invoice arrives costs the whole batch.