Design Decisions Product Managers Must Make

Here is something that gets missed in almost every conversation about RAG.

Teams treat it as an engineering problem. They spend hours debating embedding models, chunk sizes, vector databases, and retrieval algorithms. And all of that matters. But underneath every one of those technical choices is a product decision that nobody made explicitly. And that unowned decision is quietly shaping what users experience every single time they interact with the product.

The knowledge base is not infrastructure. It is product. And the decisions that define it belong to you.

Feeding everything into the knowledge base does not make the system smarter. It makes retrieval noisier and responses less reliable. Defining the scope of the knowledge base is a product decision, the same way defining the scope of a feature is.

Ask yourself before anything gets indexed:

1. What does this product actually need to know to serve users well?
2. What should be explicitly excluded, drafts, outdated versions, conflicting documents?
3. What happens when two documents give different answers to the same question?

Engineering can build the pipeline. Only you can decide what flows through it.

Every RAG system will eventually face a query it cannot find good context for. The retrieval returns thin results, or worse, results that look relevant but are not. What the product does next is entirely determined by decisions made before launch.

Does it fall back to the model's general knowledge and risk being wrong? Does it decline to answer? Does it respond with explicit uncertainty and point the user elsewhere?

There is no universally right answer. A consumer chatbot can probably fall back gracefully. A compliance tool in a regulated industry absolutely should not guess. This is a product decision disguised as a technical edge case, and it needs a deliberate answer before anything ships.