Foundations to Advanced Systems: LLMs for Product Managers
Revision
Revision: Module 3
Most AI products do not fail because they chose the wrong model. They fail because they did not understand the difference between what the model already knows and what it needs to be told.
That is the insight this entire module was built around.
Parametric knowledge is inside the model from training. Vast, capable, immediately accessible, and completely static.
Non-parametric knowledge is everything stored outside the model in a knowledge base that can be updated, audited, and maintained. RAG bridges the two, giving the model access to current, specific, and proprietary information at the exact moment a user needs it.
Embeddings are how meaning becomes searchable. Retrieval quality is what determines response quality. The knowledge base is a product that requires ownership, curation, and ongoing maintenance, not a one-time engineering setup. And the design decisions that matter most, what goes in, what happens when retrieval fails, who owns it after launch, are product decisions that need product managers to own them.
Finally, the RAG versus fine-tuning question is not a technical debate. It is a diagnostic one.
Is the model failing because it does not know something? That is a RAG problem.
Is it failing because it does not behave the right way? That is a fine-tuning problem.
Most teams reach for fine-tuning too early, before they have even properly solved the knowledge problem. Start with RAG. Introduce fine-tuning only when you have a specific behavior problem that nothing else can solve.
The model is a reasoning engine. What you put in front of it determines what comes out. This module was about learning to control that.
Quick Exercise
Find an AI product that is clearly working with a knowledge base, a support bot, a documentation assistant, or an internal tool.
Ask it three questions:
- Something that likely changed recently, a policy, a price, or a feature update
- Something phrased in an unusual or indirect way that still has a clear answer
- Something that falls just outside the obvious scope of what the tool was designed for
Then write down:
- Where you think the system retrieved its answer from, and whether it got it right
- Which failure mode from the RAG section you observed, if any
- Whether the problem you noticed feels like a knowledge problem that RAG should solve, or a behavior problem that fine-tuning might address
There are no right answers. The goal is to build the diagnostic habit of seeing AI products not as black boxes but as systems with identifiable weaknesses and addressable fixes.