Foundations to Advanced Systems: LLMs for Product Managers
Defining Quality in LLM Outputs
4.1 Defining Quality in LLM Outputs
Before you can measure quality, you have to define it. And this is where most teams stumble, not because they are not thoughtful, but because quality in LLM outputs is genuinely more complex than quality in traditional software.
When a traditional feature breaks, it breaks in a way everyone agrees is broken.
A button does not respond.
A form does not submit.
The failure is binary and obvious. LLM quality is different. A response can be fluent and wrong. It can be technically accurate and completely unhelpful. It can be safe in testing and harmful with a real user. Quality for LLMs is multidimensional, contextual, and often subjective. Which means defining it requires deliberate thought, not assumption.
The Dimensions That Actually Matter
Quality in an LLM product is not a single score. It is a profile of several dimensions, each of which matters differently depending on the use case.
Correctness is the most obvious dimension. Did the model produce a factually accurate response? For some products, like a customer support bot drawing from a knowledge base, correctness is the primary quality signal. For others, like a creative writing assistant, it matters far less.
Relevance measures whether the response actually addresses what the user asked. A model can be factually correct and still completely miss the point of a question. Relevance evaluates whether the model understood the intent behind the input, not just the literal words.
Safety measures whether the output causes harm, either through toxic content, privacy violations, or enabling misuse. In consumer-facing products, this is non-negotiable regardless of how well the product performs on every other dimension.
Tone and format may seem superficial, but they directly affect whether users trust and use the product. A response that is technically correct but written in the wrong voice for the context erodes confidence over time in ways that are hard to measure but very real.
The Hard Truth About Quality Definition
Here is what most teams skip: quality is use-case specific.
A definition of quality for a legal research assistant is fundamentally different from a definition of quality for a recipe generator.
Correctness carries different weight. Tone matters differently. The acceptable failure modes are entirely different.
This means that before any evaluation can be meaningful, the product team needs to explicitly define what good looks like for this product and these users. Not in general. Not borrowed from a benchmark. Specifically, in terms of the actual experience you are trying to create.
That definition is a product decision. It belongs to you.