Data for AI Products
The Questions to Ask in a Data Readiness Review
A 60-minute meeting, held before anyone commits to a date. It is the cheapest insurance you will ever buy.
Run this once, before the project is approved. Invite the ML lead, one data engineer, someone from ops who touches the source data every day, and someone from legal or privacy. Work through the list out loud. The goal is not to pass. The goal is to find out what you do not know while it is still free to find out.
| Area | The question | The answer you do not want to hear |
|---|---|---|
| Sourcing | Where does every row come from, and who owns that system? | "It was exported by someone who left." |
| Are we contractually and legally permitted to use this for model training specifically? | "It is our data, so yes." That is not the same question. | |
| If a customer asks us to delete their data, what happens to the model? | Silence. | |
| Quality | What does this label actually mean, and who decided? | "It is what the CRM says." |
| What is our inter-annotator agreement, per class? | "We have not measured it." | |
| Show me the twenty rows you like least in this dataset. | "They are all fine." Nobody's data is all fine. | |
| Coverage | Which customer segment is underrepresented, and by how much? | "It is representative." Ask for the breakdown by segment, market, and language. |
| Where are the rare, high-cost cases? How many examples of each? | "The model will generalise." Sometimes true. Never assume it. | |
| Evaluation | Who wrote the golden set, and does it contain cases where the right answer is "I do not know"? | "We use the standard benchmark." |
| What score gates a release, and who signs off? | "We will see how it looks." | |
| Operations | How will we know it is degrading, before a customer tells us? | "We will monitor it." Ask what, how often, and who gets the alert. |
| What is the human-in-the-loop point, and what is the fallback when the model abstains? | "There is no fallback." |
Fig 4.3 - The single most valuable outcome of this meeting is a legitimate, evidence-backed "red". It is far cheaper than a heroic amber that becomes a failed launch.
One warning about how to run this. Do not run it as an interrogation. Run it as a joint discovery, because half the time the person who knows the answer is the ops lead nobody usually invites, and they will only speak up if the room feels safe. The best data readiness reviews I have sat in ended with an engineer saying "actually, that field has meant two different things since we migrated in 2023", and everybody going quiet for a moment. That sentence is worth more than a month of sprint velocity.
YOUR MOVE THIS WEEK
Book the review. 60 minutes. Send the question table in advance so people can come with real answers rather than defensive ones. Publish the traffic-light result to your leadership, whatever colour it comes out. That transparency is how a PM earns the right to say "not yet" and be believed.