Most RAG problems are retrieval problems

When a retrieval-augmented system gives a bad answer, the first instinct is almost always to change the generation step: rewrite the prompt, raise the context limit, move to a bigger model. Occasionally that helps. Far more often the model was asked to answer from context that never contained the answer, and no amount of prompt engineering fixes a document that was not retrieved.

So before you touch the prompt, measure the ceiling.

Measure retrieval recall first

Take fifty questions you know the answers to. For each, record which document or chunk actually contains the answer. Now run only your retrieval step and ask a single question: was the right chunk in the top k results?

That percentage is the ceiling on your whole system. If recall@k is 60%, then 40% of your answers are being generated from context that cannot support them, and your best possible accuracy is 60% no matter which model you use. We have run this on systems whose owners were certain they had a hallucination problem and found recall in the fifties. The model was behaving reasonably; it was being fed the wrong pages.

If you do nothing else from this article, build the recall harness. It takes an afternoon and it tells you which half of the system to spend the next month on.

The four ways retrieval fails

When recall is low, it is nearly always one of these, in roughly this order of frequency.

1. Chunking that severs the answer

Fixed-size splitting cuts tables in half, separates a heading from the clause it governs, and strips the section number that made a paragraph interpretable. The chunk is then embedded without the context that gave it meaning, so it never matches the query.

• Split on document structure — headings, sections, list boundaries — rather than character count.
• Prepend the document title and heading path to each chunk before embedding it.
• Keep tables intact, and store them as text the model can read rather than as flattened cell soup.

2. Vocabulary mismatch between query and document

Users ask “can I get a refund after 40 days?” The policy says “returns are accepted within a 30-day window from the date of delivery.” The two share almost no lexical overlap, and depending on the embedding model they may not be close in vector space either. Meanwhile pure semantic search is famously bad at exact identifiers — part numbers, error codes, customer names — where you need a literal match.

• Run hybrid search: BM25 for exact terms, embeddings for meaning, fused with reciprocal rank fusion.
• Expand the query before searching — generate two or three paraphrases and union the results.
• For narrow domains, consider a lightweight fine-tune of the embedding model on your own question–document pairs.

3. The right chunk is retrieved, but ranked ninth

Embedding search is optimised for speed across millions of documents, not for precision at the top. It is common for the correct chunk to sit comfortably in the top 50 and never in the top 5 you actually pass to the model.

A cross-encoder reranker over the top 50 is the single highest-return change available to most RAG systems. It costs tens of milliseconds and routinely moves recall@5 by fifteen or twenty points.

4. The corpus is wrong, stale, or leaking

Sometimes retrieval works perfectly and returns a document that should never have been eligible: last year's pricing, a superseded policy, a draft that was never approved, or a record the asking user is not allowed to see.

• Filter on metadata before search, not after — effective dates, status, document type.
• Enforce permissions at query time, scoped to the requesting user. Filtering the model's output is not access control.
• Track index freshness as an operational metric with an alert, the same way you would track a stale cache.

When it really is the generation step

Once recall@k is above 90% and the answers are still wrong, you have earned the right to look at generation. The usual suspects there are a prompt that does not tell the model what to do when the context is insufficient, no citation requirement so nothing is checkable, and a context window stuffed with twenty marginal chunks that bury the two good ones.

That last one is worth stating plainly, because it is counter-intuitive: passing more context frequently makes answers worse. Precision matters more than volume. Fix retrieval, pass less, and the generation problems you were chasing tend to evaporate.