Defining Evaluation Metrics Before Shipping

This section could be the shortest in the module. But it addresses the mistake that causes the most grief, so it deserves deliberate attention.

Most teams define their evaluation metrics after something goes wrong. A user complains. A stakeholder notices a quality issue. An embarrassing response makes its way to the wrong person. And only then does the team ask: how should we have measured this?

Define your metrics before you ship. Not after. Before.

Before any LLM feature goes live, the product manager should be able to answer every one of the following:

1. What does good look like for this specific use case? Not in general. For this product, these users, these tasks.
2. What does bad look like? What specific failure modes would be unacceptable? What would constitute a serious enough quality issue to roll back the feature?
3. How will we measure each dimension of quality at scale? Human evaluation is not scalable for ongoing monitoring. What automated proxies will you use, and how confident are you that they track what actually matters?
4. What is the baseline? If you cannot measure whether things are improving, you cannot improve them. Establish a baseline before any changes are made.
5. What triggers a response? If a quality metric drops below a certain threshold, what happens? Who is notified? What is the process for investigation and remediation?

The discipline of answering these questions before shipping is what separates teams that manage quality intentionally from teams that manage it reactively, which is another way of saying after the damage is done.