Performance Optimization

Speed is a product quality dimension. Not a nice-to-have. Not an engineering concern. A product quality dimension that users experience on every single interaction, even when they never consciously think about it.

The research on this is consistent. Users tolerate seconds of waiting during initial load. They do not tolerate seconds of waiting for an AI feature that is supposed to feel intelligent and responsive. The expectation is different. And meeting that expectation in production, consistently, under real load, requires deliberate design.

Before optimizing performance, you need to measure it. And in LLM systems, there are specific metrics that map directly to user experience in ways that general latency numbers do not capture.

1. Time to First Token (TTFT) is the time between a user sending a message and the first word of the response appearing on screen. This is the metric users feel most acutely. A product that streams its response and starts appearing in under a second feels fast, even if the full response takes five seconds. Time to first token is influenced by request queuing, prefill computation, and network latency. Users feel this as the initial delay before output begins, and it is the most perceptible performance dimension in interactive applications.

1. Total response time is how long the full response takes. For short responses, this is nearly the same as TTFT. For long responses, streaming makes this less critical because users are already reading before it completes.

1. P95 and P99 latency matter more than average latency for understanding the user experience. An average response time of one second means nothing if five percent of users are waiting ten seconds. Those users exist, they have a bad experience, and they form impressions about the product based on that experience.

Performance optimization in LLM products operates at several levels, and the highest-impact ones are often product decisions, not engineering decisions.

1. Streaming responses is the single most impactful change most products can make to perceived performance. Rather than waiting for the full response to generate before displaying anything, streaming sends tokens to the user as they are generated. This is a product and UX decision as much as a technical one.

1. Prompt optimization directly reduces latency. A shorter, more focused system instruction processes faster than a long, unfocused one. Retrieved context that is precisely relevant processes faster than context that dumps everything and hopes the model finds what it needs. Every token in the prompt is compute. Reducing unnecessary tokens is both a cost and a performance improvement.

1. Caching is appropriate for use cases where the same or similar queries are asked repeatedly. Caching results for common queries and precomputing answers for high-frequency, low-variability requests reduces redundant computation and delivers faster responses for the most common user interactions. For a product where a significant fraction of queries are predictable, caching can make those interactions feel nearly instantaneous.

1. Right-sizing the model for the task is a performance optimization as much as a cost one. A smaller model that handles a task adequately will always respond faster than a larger model handling the same task with unnecessary overhead. The question is always: what is the minimum capability required for this specific task to serve this specific user well?