How Much Data Do You Actually Need? (Setting Realistic Expectations)

Far less than you fear for style, far more than you hope for knowledge, and the difference is the whole lesson.

This is the question every executive asks in the first meeting, and the honest answer is "it depends on what you are trying to teach". So let us make "it depends" useful by splitting it into the two things data can teach a model.

• Behaviour. Format, tone, structure, how to reason through your kind of problem, when to refuse. This is cheap. Modern base models already know how to write, they just do not know how you want them to write.

• Knowledge. Facts about your product, your pricing, your policies, your customers. This is expensive to teach through training and is usually the wrong tool for the job. Module 3 explains why.

Now the numbers, from primary sources rather than vibes.

OpenAI's supervised fine-tuning documentation states the practical floor plainly: a fine-tuning job requires at least 10 examples, clear improvements typically appear with 50 to 100 examples, and they recommend starting with 50 well-crafted demonstrations and evaluating before scaling up. Read that again if you have ever been told a fine-tune needs a hundred thousand rows to be worth doing.

Then there is the study that reframed the whole conversation. Meta's LIMA paper fine-tuned a 65B parameter model on just 1,000 carefully curated prompt and response pairs, with no reinforcement learning and no preference modelling. In a controlled human study, LIMA's responses were rated equal to or better than GPT-4's in 43% of cases, better than Bard in 58%, and better than DaVinci-003 in 65%. The paper's conclusion is the sentence to quote in your next planning meeting: nearly all of a model's knowledge is acquired during pretraining, and only a limited amount of instruction data is needed to teach it to produce high-quality output.

Fig 2.3 - Most product teams should live in the bottom three rungs. If someone proposes the top rung, ask what business problem requires it.

Three planning rules follow from this, and they are the ones to hand to your leadership.

3. Never start at the top of the ladder. Get a prompt working first. It is the cheapest possible experiment and it produces something equally valuable: a clear list of the specific ways the model fails, which is exactly the list your training data should target.

4. Buy quality before quantity. 200 examples written by your best support agent will beat 5,000 scraped from a CRM where "resolved" meant "customer gave up". Curation is a higher-leverage investment than collection.

5. Budget the evaluation set first. Teams routinely spend everything on training data and label the test set in a rushed afternoon. That is backwards. Your evaluation set is the only instrument you have for knowing whether anything worked. Spend expert money there, on 200 to 500 examples, before you spend crowd money on the training set.

THE ANSWER TO GIVE YOUR CEO

"For tone, format, and consistency, we need a few hundred excellent examples and about three weeks. For factual accuracy about our own product, more training data is the wrong lever entirely. We need retrieval over a curated knowledge base, and the work there is content operations, not machine learning."

YOUR MOVE THIS WEEK

Write 20 gold-standard examples of your feature's ideal output by hand. Time yourself. Multiply by 10 to estimate 200. That number, in hours, is your real data budget, and it is now defensible instead of invented.