Retrieval-augmented generation, honestly

The demo went well. Twelve questions from the room, twelve answers, each with a document name beside it. The head of operations asked the one question she cared about and got the right policy quoted back at her. The project was approved that afternoon.

Six weeks after launch, a support agent followed an answer that quoted the refund window from a policy replaced in 2023. The superseded document was still in the shared drive. It was still in the index. It ranked first because it was longer, more explicit, and used the exact words of the question, while the current policy referred to the same rule in one indirect sentence.

Nothing in that failure is about the language model. Every part of it happened before the model was asked anything.

What retrieval is actually for

A model's weights encode the text they were trained on, and nothing else. The argument in Language models predict the next token covers why: the objective rewards predicting continuations, so knowledge is stored diffusely, cannot be updated without retraining, and cannot be traced to a source.

Lewis and colleagues framed the fix precisely. Pre-trained models with a differentiable access mechanism to explicit non-parametric memory, they argue, can overcome the limits of parametric-only approaches. Their system combines pre-trained parametric memory, a sequence-to-sequence model, with non-parametric memory, a dense vector index of Wikipedia accessed with a pre-trained neural retriever. They report state-of-the-art results on three open-domain question answering tasks and, for language generation, more specific, diverse and factual language than a state-of-the-art parametric-only baseline.

Two words in that description carry the value. Explicit, because the knowledge sits in a store you can inspect, edit, and delete. And non-parametric, because updating it costs one write rather than a training run.

So retrieval fixes exactly this: the model does not know your documents, cannot be updated cheaply, and cannot cite. It does not fix reasoning, arithmetic, judgement, or the tendency to produce an answer of the correct shape when nothing supports one.

The path, and where the answer is really decided

Read the drawing from the top left, not the bottom. Most of a retrieval system runs before anyone asks anything: documents are cut into chunks, each chunk is embedded using the machinery in Embeddings are coordinates for meaning, and the vectors go into an index.

At question time the query is embedded with the same model, the index returns its nearest neighbours, and the top handful are pasted into a prompt with the question.

The consequence that changes how you debug: the model never sees your corpus. It sees five passages. If the answer was in the passage that ranked sixth, no property of the model can recover it. Retrieval quality is a ceiling on system quality, and it is measurable separately, which is the single most useful thing to know about these systems.

Dense retrieval is not automatically better than keyword search

The vector-first instinct is worth arguing with, because the evidence is more mixed than the marketing.

Karpukhin and colleagues made the case for dense retrieval carefully. They showed that retrieval can be implemented using dense representations alone, with embeddings learned from a small number of questions and passages by a dual-encoder framework, and their dense retriever outperformed a strong Lucene-BM25 system by 9 to 19 percent absolute in top-20 passage retrieval accuracy across open-domain question answering datasets.

That is a real and large gain. It is also a gain measured on the kind of data the retriever was trained for.

Thakur, Reimers, Ruckle, Srivastava and Gurevych built a benchmark specifically to test what happens outside that setting. Neural retrieval models, they note, have often been studied in homogeneous and narrow settings, which limits insight into their out-of-distribution generalisation. Their benchmark covers 18 publicly available datasets across diverse retrieval tasks and domains, evaluated over 10 retrieval systems spanning lexical, sparse, dense, late-interaction and re-ranking approaches. Their finding is the one to remember: in the zero-shot setting dense retrieval models often underperform, with considerable room for improvement in generalisation, while BM25 holds up across domains and re-ranking and late-interaction approaches perform best at higher computational cost.

Which is to say: a dense retriever tuned on question answering, dropped onto your product documentation, may well lose to a keyword index you could have built in an afternoon. The practical answer in most production systems is to run both and combine the rankings, then rerank the union with a cross-encoder, for the reasons costed out in the previous article.

Position inside the prompt changes the answer

Suppose retrieval works. The right passage is in the top five. You are still not done.

Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni and Liang tested how well models use the context they are given, on multi-document question answering and key-value retrieval. Performance, they found, can degrade significantly when the position of the relevant information changes, which indicates that current models do not reliably use information across long input contexts. Specifically, performance is often highest when the relevant information occurs at the beginning or end of the input context, and degrades significantly when the model must access information in the middle of long contexts, even for models built for long contexts.

Sit with the engineering consequence. Your system retrieves ten passages and concatenates them in rank order. The correct one ranks fourth, so it lands in the middle, which is the worst position in the prompt. Retrieval succeeded, the evaluation harness marks it correct on recall, and the answer is still wrong.

It also punctures the standard escape route. "Just put more in the context" is not free, and the extra passages push the good one towards the middle while adding cost on every request.

The five failures that account for most of the trouble

Chunking that cuts through the answer. A policy states a rule in one sentence and its exception two paragraphs later. Split them across chunks and any single retrieved chunk is misleading rather than incomplete, which is worse. Chunk on document structure, keep headings attached to the text under them, and overlap where sections are dense.

Stale and superseded documents. The failure this article opened with. A vector index has no concept of a document being replaced; a deleted file often survives in the index because deletion was never wired up. Store an effective date on every chunk, filter by it, and treat index deletion as a first-class operation with a test.

Near-duplicates eating the slots. Four of your five slots go to four copies of the same onboarding document that differ only in a date in the footer. Deduplicate on content hash at ingest, and diversify results at retrieval.

Questions that are not lookups. "How many refunds did we issue last quarter" cannot be answered by finding a passage. Nor can "compare our policy with theirs". Retrieval answers a question when the answer exists as text somewhere. Aggregation, counting, and comparison need a different tool, which is what the next-but-one article on agents is really about.

No abstention. The most damaging default in the whole pattern is a system that always answers. When the top result scores 0.31 and nothing in it addresses the question, the correct behaviour is to say so. Designing for that is the subject of Hallucination is a design problem.

How to know whether it works

Evaluate the two halves separately, because they fail differently and one masks the other.

For retrieval, build a set of real questions with the passages that genuinely answer them marked by a person. Then measure whether the right passage appears in the top k, and at what rank. This is an ordinary information retrieval evaluation and everything in Train, test, and the lie of a single score applies: one aggregate number over a friendly question set will flatter you.

For generation, given the correct passages, check whether the answer is supported by them. That is a different question from whether the answer is true, and it is the one you can actually hold the system to.

Track a third number that most teams never write down: how often the system should have declined. Sample real traffic, have a person mark the questions your corpus cannot answer, and measure what the system did with them. That number moves first when a corpus goes stale, and nothing else you monitor will catch it.

What this pattern is worth

Retrieval turns a model that cannot cite into a system that can, makes knowledge editable by writing a document rather than by training, and lets a reader check the answer against the source. Those are large, durable benefits and they are why the pattern became standard within a year of the paper naming it.

What it does not do is make a system trustworthy on its own. The demo that answered twelve questions was not evidence that retrieval worked; it was evidence that twelve questions were easy. The work is in the ranking, the freshness, the chunk boundaries, and the decision to stay silent.

The next article steps back to a question every team asks in week two: whether any of this should have been a fine-tuned model instead.

References

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, Sebastian Riedel and Douwe Kiela, arXiv, 2020.
2. Dense Passage Retrieval for Open-Domain Question Answering. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen and Wen-tau Yih, arXiv, 2020.
3. BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models. Nandan Thakur, Nils Reimers, Andreas Ruckle, Abhishek Srivastava and Iryna Gurevych, arXiv, 2021.
4. Lost in the Middle: How Language Models Use Long Contexts. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang, arXiv, 2023.