Cost Architecture & Unit Economics

Here is a number that surprises most product managers the first time they see it.

A single API call to a large language model, with a moderately complex prompt and a detailed response, can cost anywhere from fractions of a cent to several cents, depending on the model and the token count. That sounds trivial. At a hundred calls per day, it is. At a million calls per day, it is not. At ten million calls per day, it is a major line item on the company's P&L.

Cost in LLM products is not a fixed overhead. It is a variable that scales directly with usage. And understanding the economics of that variable before it surprises you is one of the most practically important skills a product manager can develop.

Every interaction with an LLM API is priced in tokens. Input tokens, the system instruction, retrieved context, conversation history, and the user's message, and output tokens, what the model generates. Both sides of that transaction cost money, and the input is often far larger than people realize.

The cost drivers are straightforward once you understand them:

1. Model choice: Larger, more capable models cost significantly more per token than smaller, faster ones. A task that genuinely requires a frontier model is worth the cost. A task that a smaller model handles equally well should not be sent to the expensive one by default.
2. Prompt length: A long system instruction, retrieved documents from RAG, and full conversation history can add thousands of tokens to every single call. At scale, unoptimized prompt length is one of the biggest sources of unnecessary cost.
3. Output length: Verbose responses cost more. Capping output length where appropriate and designing prompts that encourage concise responses has a direct and measurable impact on cost.
4. Call frequency: Some features make one model call per user action. Others make three or four, chained together. Each additional call in a chain multiplies the cost of every user interaction.

Aggressive caching and precomputation reduces redundant API calls significantly. Using Redis to cache results for common queries and precomputing answers for high-frequency, low-variability requests represents one of the most impactful cost optimizations available to production teams.

Cost in isolation is not meaningful. What matters is cost relative to value delivered. This is the concept of unit economics, and developing a clear model for it is one of the things that distinguishes mature AI product teams from teams that discover their cost problem only after it is a crisis.

The questions worth asking before launch:

1. What is the cost per user session for this feature, and how does it scale with usage patterns?
2. What is the cost per successful outcome, where successful means the user actually got what they needed?
3. At what usage level does the current architecture become unsustainable, and what needs to change before that point?
4. Are there usage patterns that would generate disproportionate cost relative to value, and should those be bounded or priced differently?