The Next RAG Problem Isn’t Retrieval — It’s Knowing When Not to Retrieve

The most expensive RAG failure is often not an empty search result. It is a confident answer built on chunks that should never have been retrieved in the first place. A user asks, “What is my order status?” The system retrieves three generic shipping policy documents, one old FAQ, and a refund guide. The model synthesizes something plausible, cites the wrong policy, and now support has to clean up the mess. Retrieval worked. The pipeline still failed. This is the next RAG problem: not just finding relevant documents, but deciding whether retrieval should happen at all. Production RAG systems are moving beyond the simple “retrieve everything that looks similar” pattern. The harder design question is: Should this request use retrieval, call a tool, ask a clarifying question, answer directly, or refuse? That decision affects correctness, latency, cost, security, and user trust. TL;DR Retrieval is not always the right first step. Some questions need live tools, not docu