Data for AI Products
When Your Data Determines the Model Choice, Not the Other Way Around
There is research on this. Use it, so the argument stops being about who is more senior.
The RAG versus fine-tuning debate gets settled in most companies by whoever speaks with the most confidence. That is a bad way to spend a quarter. Fortunately there is published evidence you can bring to the table.
Ovadia and colleagues published "Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs" (EMNLP 2024). They tested both approaches on knowledge-intensive tasks, and the finding was consistent: RAG outperformed unsupervised fine-tuning, both for knowledge the models had seen in training and for entirely new knowledge. On a task built from current events, which is the purest test of "can you learn a new fact", RAG scored around 0.875 while fine-tuning scored around 0.504 for the same base model. That is not a marginal difference. That is a different product.
Fig 3.2 - Note what this does and does not say. Fine-tuning still wins on style, format, and task behaviour. It loses on facts.
The same paper found something subtler and more useful: models struggle to absorb new factual information through fine-tuning unless they are exposed to many paraphrased variations of the same fact. Think about the cost implication. To teach a fine-tuned model that your refund window is now 14 days, you would need dozens of differently worded examples of that fact, and you would need to do it again the next time policy changes. Or you could edit one line in a document that a retrieval system reads. That is the entire argument, in one line of your budget.
So here is the data-first decision procedure. Work down this list in order and stop at the first honest "no".
| Ask | If yes | If no |
|---|---|---|
| Do we have a clean, current, authoritative body of documents? | RAG is on the table | RAG will fail. Fix the content first. This is a content-ops project, not an ML project. |
| Do we have at least a few hundred examples of the exact input-output behaviour we want? | Fine-tuning is on the table | You cannot fine-tune. You can only prompt, or go create the examples. |
| Do the facts change more than once a quarter? | Retrieval, not weights | Fine-tuning becomes viable |
| Must we show the user a source? | Retrieval, mandatory | Free choice |
| Is the failure about consistency, format, or refusing correctly? | Fine-tuning earns its cost | Prompting is probably enough |
| Is latency or per-token cost the binding constraint at our volume? | A small fine-tuned model can beat a large prompted one on both | Stay with the big model and a good prompt |
That last row is the legitimate, underrated reason to fine-tune, and it is a business reason rather than a quality one. A small fine-tuned model that matches a large prompted model's quality on your narrow task, at a fraction of the cost and latency, is a real win at scale. But notice the order. You establish the quality bar with the big model first, then you try to hit it more cheaply. Doing it the other way round means you are optimising a target you have not defined.
YOUR MOVE THIS WEEK
Put this sentence in your next architecture review: "Before we discuss the model, can we look at the data we would need for each option, and confirm we actually have it?" Half the time the meeting ends there, correctly.