Tokens, Context Windows, and Generation Controls

To work effectively with LLM's as a product builder, you need a working understanding of three foundational mechanics:

1. How text is broken down into tokens
2. How much text a model can hold in its attention at once
3. How you can influence the style and nature of what the model generates.

These are not JUST technical details. They directly affect product decisions around feature design, cost, reliability, and user experience.

There is one more layer to understand about the prediction mechanism and it matters more than it might initially seem. LLMs do not actually process words. They process tokens.

A token is a chunk of text, roughly three to four characters on average in English. Common words are often a single token.

Less common words, technical terms, or words in other languages may be split across multiple tokens.

This matters for product design for a few reasons :

Token limits define how much text a model can process in a single interaction, costs in most LLM APIs are calculated based on token count.

Tasks like processing long documents or maintaining extended conversations, require deliberate thinking about how to stay within these boundaries without losing important context.

Understanding the token-level mechanics gives you a clearer mental model of what is happening under the hood, and helps you make better decisions when designing features, setting context windows, and estimating the cost of running LLM-powered functionality at scale.

The context window is the total number of tokens a model can process at one time.

This includes everything: the system instructions you have provided, the conversation history, any documents or data you have passed in, and the response the model is generating.

Once the combined token count exceeds the context window, the model can no longer access the earlier parts of the conversation. Think of the context window as the model's working memory. Unlike a human who can vaguely recall a conversation from last week, the model has no memory outside of what is currently in the context window. Anything outside that window simply does not exist for the model in that interaction.

For product designers, this creates real constraints. Long conversations will eventually push early messages out of the window. Large documents passed as context consume a significant portion of available tokens. Features like chat history, document analysis, or multi-step workflows all require deliberate decisions about what to keep in context, what to summarize, and what to drop entirely.

Context windows have grown significantly over recent years, from a few thousand tokens to hundreds of thousands in some models. But even with large windows, efficiency matters, because every token in the window contributes to latency and cost.

Beyond what you put into the model, you also have control over how the model generates its output.

The most important of these controls are Temperature, Top-p, and Max tokens.

Temperature is the most commonly used control. It governs how much randomness the model introduces when selecting the next token. A low temperature, close to zero, makes the model conservative and deterministic, it will almost always pick the most statistically likely next token. A high temperature introduces more variation and creativity, but also more risk of incoherence.

For product use cases that require precision, such as data extraction or classification, you generally want a low temperature. For creative or generative tasks, a higher temperature produces more varied and interesting results.

Top-p, sometimes called nucleus sampling, is an alternative way to control randomness. Rather than adjusting the probability of all possible tokens equally, it limits the model to choosing from only the most probable tokens that together make up a set percentage of the total probability. In practice, temperature and top-p are often used together, though many teams find that tuning one at a time is easier to reason about.

Max tokens simply sets a ceiling on how long the model's response can be. This is useful for controlling costs, enforcing consistent response formats, and preventing the model from generating unnecessarily long outputs in contexts where brevity is required.