Foundations to Advanced Systems: LLMs for Product Managers
What Actually Happens in a RAG System
3.2 What Actually Happens in a RAG System
RAG stands for Retrieval-Augmented Generation. It is not just about retrieving documents. It is about augmenting the model's knowledge in real time, so that every response is grounded in information that is current, specific, and verifiable.
Here is how the pipeline actually works.
Building the Knowledge Base
Before any user query is processed, your documents need to be prepared. This happens in three steps.
Loading
This means ingesting your source material, PDFs, internal wikis, product docs, policy documents, whatever your product needs to know about.
Chunking
This means breaking those documents into smaller pieces. This matters more than it sounds. Chunks that are too large bring in too much noise. Chunks that are too small lose important context. The right size depends on the nature of your documents and the types of questions your users will ask.
Embedding
This means converting each chunk into a numerical representation of its meaning, called a vector. These vectors are stored in a vector database, a specialized store optimized for finding similar meanings quickly. We will cover embeddings in depth in the next section.
Retrieval and Generation at Query Time
When a user sends a message, the system converts that message into a vector and searches the knowledge base for the chunks most semantically similar to what the user is asking. The most relevant chunks are retrieved and assembled with the user's query into a single, enriched prompt that goes to the model.
The model now has two inputs: its training knowledge and the retrieved context. When the model can choose between parametric and non-parametric information, it relies more on the retrieved context than on its internal knowledge. This is what makes RAG so powerful. Provide high-quality retrieved context and the model anchors its response to it, producing answers that are current, specific, and grounded.
The practical implication is critical: the quality of a RAG system is roughly 80% retrieval and 20% generation. If your retriever surfaces irrelevant documents, even the smartest model produces a bad answer. Most teams optimize for the wrong 20%.