Why most RAG systems fail at retrieval, not generation

When a retrieval-augmented assistant gives a bad answer, the instinct is to fix the prompt. In our experience across several production systems, including a self-hosted cluster with more than 55 million objects, the prompt was rarely the problem. The retriever was.

The test that changed our process

Before touching a prompt, we now build a golden set: a few hundred real questions paired with the passages that should answer them. We measure recall at k for the retriever alone. On a new corpus it is common to see recall below 60% on the first pass, which means the model is being asked to answer four questions in ten without the evidence in front of it. No prompt fixes that.

What actually moved recall

Three things, in order of impact. First, chunking: overlapping chunks aligned to document structure (headings, clauses, table rows) beat fixed-size windows by a wide margin. Second, hybrid search: combining dense vectors with a keyword index recovered exact identifiers, product codes and names that embeddings blur. Third, re-ranking a larger candidate set with a cross-encoder before passing the top few to the model.

Quantisation is a cost decision, not a quality decision

On the large cluster we applied RQ-8 quantisation and measured roughly 4x compression at 85–92% recall, depending on the collection. The variance between collections is the point: measure per collection and decide per collection. Some deserve full-precision vectors; most do not.

Freshness is retrieval too

An index that is a day old returns yesterday's policy. A change-data-capture pipeline that applies only deltas kept our index minutes behind the source. Stale evidence looks exactly like hallucination to the person reading the answer.

The checklist we use

Golden set built. Recall at k measured before and after every retrieval change. Hybrid search on. Re-ranker evaluated. Quantisation measured per collection. Freshness SLA defined. Only then: prompts.