RAG Should Not Be the Default Answer

Retrieval-augmented generation is the current default for any workflow that touches text. It shouldn’t be.

RAG is the right answer when the input is genuinely fuzzy: natural language, partial matches, semantic similarity. RAG is the wrong answer when the input is structured, the lookup is deterministic, or the output needs to be exact.

Using RAG where deterministic logic would do gives you slower, more expensive workflows that are harder to test and harder to fix when they go wrong.

Here is what that looks like. A team used similarity search to answer “what is the status of my order” because the question arrived as text. A customer asked about order 4471. The retrieval returned the nearest match, order 4417, and the model confidently paraphrased its status. The customer was told a delivered order was still processing. One transposed digit; similarity search does not care about digits. A deterministic lookup on the order ID would have been exact, free, and impossible to get subtly wrong.

When RAG fits, and when it does not

RAG earns its cost only when the input is genuinely fuzzy: a natural-language query against unstructured material (PDFs, email threads, support tickets) where the closest passage is more useful than an exact row and the output is a synthesis rather than a lookup. The moment the input is structured (a customer ID, an order number, a SKU) and the answer must be exact and auditable, RAG is the wrong tool and a deterministic lookup is the right one. The order-status failure was precisely that line crossed: a structured ID handed to similarity search, which answered with the closest thing instead of the correct thing.

The privacy axis (the second question)

RAG requires sending data to an embedding model. Deterministic logic doesn’t.

If the data is confidential (payroll, health records, internal financials), that’s a separate reason to default to deterministic. RAG is fine for public docs, support knowledge bases, and content. Not for the things you’d never paste into a public LLM chat.

The decision

Three questions settle it for any text-touching workflow: is the input structured or fuzzy, is the lookup exact or approximate, and is the data confidential. Anything structured or exact points to deterministic. Confidential settles it on its own, because RAG means sending the data to an embedding model in the first place. The only combination where RAG is the fit is fuzzy, approximate, and public. Most real workflows are a mix, and the honest answer is a hybrid: a deterministic lookup to fetch the exact record, RAG only on the unstructured part of the result, never on the part that already has a correct answer.

The rule

RAG is a tool for one job: turning genuinely fuzzy language into a synthesized answer. It is not a default. For a structured input and an exact lookup, deterministic logic is faster, cheaper, testable, and it cannot return order 4417 when you asked for 4471.

Default to deterministic. Reach for RAG only when the input is actually fuzzy and the data is not confidential.