Why Your AI Feature Is Only as Good as Its Data

The model is a commodity. Your data is not. That is the whole strategic point.

Every team you compete with can call the same frontier models you can. The weights are not your moat. Your prompt is copyable in an afternoon. What is not copyable is the specific, messy, hard-won record of how your customers actually behave and how your best people actually respond to them. That is the asset. Everything else is plumbing.

This is not a motivational statement, it is an empirical one. The most useful study on this for PMs is a Google research paper from CHI 2021 with a title that should be printed on every AI team's wall: "Everyone wants to do the model work, not the data work". The researchers interviewed 53 AI practitioners working on high-stakes systems in India, the United States, and several African countries. They found that 92% had experienced at least one "data cascade", which is their term for a data problem upstream that quietly compounds into a large downstream failure. Around 45% had experienced two or more in a single project.

The word "quietly" is the important one. Data cascades are usually invisible until launch, because there is no dashboard for them. The model trains fine. The offline metric looks fine. Then real users arrive and the thing falls over.

Those second and third numbers come from another study every AI PM should know about. Curtis Northcutt and colleagues at MIT ran an error analysis across ten of the most commonly used benchmark datasets in machine learning. Not obscure ones. The canonical ones. They found an average of 3.4% label errors, including roughly 6% in the ImageNet validation set. You can browse the actual mislabelled examples at labelerrors.com, and I recommend you do, because seeing a picture of a frog labelled as a cat does more for your intuition than any lecture.

Here is the part that should change how you run reviews. The same paper showed that once label noise crosses a modest threshold, the ranking of models flips. On corrected ImageNet labels, the smaller ResNet-18 outperforms the bigger ResNet-50 once the prevalence of originally mislabelled examples rises by just 6%. Translated into product language: if your evaluation set is dirty, your team will confidently pick the wrong model, and the metric will tell them they were right.

Fig 1.2 - Stylised from Northcutt et al. (2021). Past a threshold, the larger model's apparent advantage disappears and the ranking inverts. The models did not change. The labels did.

So when your ML lead says "we tried three models and picked the best one", the correct follow-up question is not "which one" and it is definitely not "can we try a bigger one". It is: "how clean is the set we scored them on, and who checked it?"

WHERE THIS BITES IN REAL PRODUCTS

Support automation. Trained on resolved tickets. But "resolved" in the CRM often means "customer stopped replying". The model learns to close conversations, not to solve problems. Lead scoring. Trained on historical conversions. But historically your sales team only called leads from three cities. The model learns those three cities are where the good leads are, because it never saw evidence of anything else. Content moderation. Trained on moderator decisions. Moderators disagree with each other roughly as often as they agree on the hard cases. The model learns the average of that disagreement, which is a position no human actually holds.

Each of those failures is a product failure with a data cause. And each of them would have been caught by a PM who asked one boring question early: what exactly does this label mean, and who decided it?

YOUR MOVE THIS WEEK

Ask your ML team for 30 random rows from the training set and 30 from the evaluation set. Read them yourself. Do not delegate this. You are looking for one thing only: examples where you, the person who knows the customer best, would have given a different answer than the label says. If you find more than two or three, you do not have a modelling problem, you have a data problem, and you have just saved the quarter.