RAG Isn't Magic: When Retrieval Helps and When It Hurts

RAG, retrieval-augmented generation, gets talked about like a cure. Bolt a vector database onto your model, let it look things up, and suddenly the AI knows everything about your business. That's the pitch. The reality is more useful and less magical: RAG is a tool that's excellent for some jobs and actively harmful for others, and knowing the difference saves you a lot of wasted effort.

Here's the plain version of what it does. Instead of relying only on what the model already knows, you store your information in a way the system can search, and at question time it retrieves the relevant pieces and hands them to the model along with the question. Done well, it grounds answers in your actual data. Done reflexively, it introduces a whole new way to be wrong.

When RAG genuinely helps

RAG earns its keep when you have a large body of knowledge that changes over time and is too big to stuff into every prompt. Think a sprawling, frequently-updated documentation set, a knowledge base of thousands of support articles, a library of contracts or policies. You can't fit it all into context, and you don't want to rebuild anything every time it changes. Retrieval lets the agent reach into that store and pull only what's relevant to the question in front of it.

That's the sweet spot: big, dynamic, and you need the answer grounded in a specific source rather than the model's general knowledge.

When RAG hurts

RAG starts causing problems in a few common situations.

Your context is small and stable. If the information the agent needs fits comfortably in the prompt and rarely changes, retrieval adds machinery and failure modes for no benefit. Just put it in the context directly. A retrieval step you didn't need is a retrieval step that can fetch the wrong thing.

Retrieval pulls the wrong chunks. RAG is only as good as what it retrieves. If your data is poorly structured or chunked badly, the system confidently hands the model irrelevant or misleading passages, and the model answers based on them. Now you have a wrong answer that looks well-sourced, which is harder to catch than an obvious guess.

The question needs the whole picture, not a lookup. Some questions aren't "find the relevant passage." They need the agent to synthesize across many things at once. Retrieval that returns a few chunks can actively get in the way of that.

The real lesson

RAG isn't the goal. A correct, grounded answer is the goal, and RAG is one way to get there when the shape of the problem fits. The teams that struggle treat it as a default to reach for, then spend months debugging retrieval quality when the actual issue is that their data was never in good enough shape to retrieve well. Which is the usual story: the problem is the data, not the technique.

Deciding whether you even need retrieval, and building it so it helps instead of hurts, is part of designing a context layer that holds up. It's exactly the kind of judgment the 30% in the 70/30 Method exists for, and it's the Agentic OS work I do with clients. If someone's told you to "just add RAG," it's worth a second opinion first. Let's talk.