Hallucination is a design problem

Picture the screen a user is actually looking at. A four-sentence answer in the same typeface as the rest of the product. Underneath it, three small chips carrying document names. The chips are grey, rounded, identically sized, and each one is a link.

The first chip points at a page that says exactly what the answer says. So does the second. The third points at a document that discusses the topic and does not contain the claim attributed to it.

A person reading that screen has no way to tell the three apart without opening all three, which nobody does. The product designed a control that presents supported and unsupported claims in identical visual language, and then relied on the user to distinguish them.

That is the argument of this article. The generation failure is real, well studied, and only partly fixable. The trust failure sits in the interface, and the interface is entirely under your control.

The measurement worth arguing from

Liu, Zhang and Liang audited this directly rather than reasoning about it. Verifiability, they argue, is a prerequisite for a trustworthy generative search engine: systems should cite comprehensively, so that all statements are fully supported by citations, and accurately, so that every citation supports its associated statement. They ran a human evaluation across four popular generative search engines using a diverse set of queries drawn from historical search queries and open-ended questions collected from discussion sites.

Their finding, verbatim in shape: responses from existing generative search engines are fluent and appear informative, but frequently contain unsupported statements and inaccurate citations, with on average a mere 51.5 percent of generated sentences fully supported by citations and only 74.5 percent of citations supporting their associated sentence.

Roughly half the sentences in a cited answer were not actually supported by what was cited, in shipped products, audited by people. And the citation chips were rendered the same way regardless.

Note the phrase "fluent and appear informative", because it names the mechanism. Fluency is free, for the reason given in Language models predict the next token: the training objective rewards plausible continuations, and a plausible continuation of a citation is a citation-shaped thing. Nothing in the objective distinguishes a supported sentence from an unsupported one, so nothing in the output does either.

Grounding helps, and does not finish the job

The instinct is to ground harder: retrieve better passages, instruct the model to use only them. That is correct and it is the right first move, but it does not close the gap on its own.

Maynez, Narayan, Bohnet and McDonald established this on the cleanest possible version of the task, summarising a single document that is right there in the input. They analysed neural abstractive summarisation systems and found the models highly prone to hallucinate content that is unfaithful to the input document, with their human annotators finding substantial amounts of hallucinated content in all model generated summaries.

All of them. With the source document in the context. Grounding constrains what the model tends to say; it does not constitute a check that what it said follows from the source.

The same paper points at the check. They show that textual entailment measures better correlate with faithfulness than standard metrics such as ROUGE, potentially leading the way to automatic evaluation metrics as well as training and decoding criteria. In production terms: score each generated sentence against the passage it claims to come from, with a model asked one narrow question, and treat the score as a property of that sentence rather than of the answer. That is exactly the per-property grading argued for in Evaluating a system that can say anything, and Zheng and colleagues' finding that a strong model judge reaches over 80 percent agreement with human preferences, the same level as between humans, is what makes running it on every sentence affordable.

Eighty percent agreement is not a certification. It is enough to sort sentences into two piles, and two piles is all the interface needs.

What the interface should do with two piles

Once each sentence carries a support verdict, the design work begins, and this is where most products stop short.

Show the supporting text, not a link to a document. A chip that says Refund policy v4.pdf asks the user to go and verify, which is a task, and the user will not do it. A hover or an inline expansion that shows the two sentences the claim came from is a glance. The cost of checking has to be under a second or checking does not happen.

Draw unsupported sentences differently. If your entailment check cannot confirm a sentence, that sentence should not look like the ones it can confirm. It does not need alarm; it needs a visible difference and a plain label. Vorvoreanu, Amershi and Collisson put this second in their list of eighteen guidelines, right after making clear what the system can do: make clear how well the system can do what it can do. A uniform presentation of mixed-confidence output violates that at the level of the pixel.

Make declining a real output with its own layout. Most systems can only answer. When retrieval returns nothing relevant, the model is still asked to produce prose, and it produces prose. Their tenth guideline is the fix stated as design advice: scope services when in doubt. The engineering version is a threshold on retrieval score and support coverage, below which the product returns something structurally different: what it looked in, what it found, and what it would need in order to answer. That screen needs to be designed properly, because a badly designed refusal teaches users to rephrase until they get prose, which is the worst outcome available.

Make correction cheap. Guidelines eight and nine are support efficient dismissal and support efficient correction. A user who spots the wrong policy should be able to say so in one click, from that sentence, with the sentence and its retrieved passages attached to the report. That single control is also the cheapest source of evaluation data you will ever build.

Say why. Guideline eleven is make clear why the system did what it did. For a retrieval system this is unusually easy and unusually often skipped: show which passages were retrieved and which were used. When the answer is wrong, the user immediately sees whether it was retrieval or generation, and so do you.

Anti-patterns that manufacture trust

Some common choices actively add confidence the system has not earned.

Word-by-word streaming as theatre. Streaming genuinely improves perceived latency and belongs in the product for the reasons in Putting a model behind an interface. It also reads as thinking, and it makes an answer feel considered in proportion to how slowly it appears. Stream, but do not slow it down, and do not stream a canned unsupported answer with the same animation as a verified one.

A face, a name, a personality. An avatar and a first-person voice recruit social instincts that have nothing to do with accuracy. Their fifth guideline is match relevant social norms, which is not an instruction to be charming; a system that presents as a knowledgeable colleague is making a claim about its reliability in a channel the user cannot audit.

Uniform prose for mixed evidence. The default failure. One paragraph, four sentences, three well supported and one invented, all in the same font at the same weight.

Hedging language as a substitute for a confidence signal. "It appears that" and "based on the documents" cost nothing to generate and correlate with nothing. Users learn to ignore them within a week, and they leave the real signal unexpressed.

Apologising instead of stopping. A system that says sorry, then produces another confident wrong answer, has converted an error into two errors and taught the user that the apology is decorative.

What to measure, and who owns it

Two numbers belong on the same dashboard as your latency and cost.

Support coverage: the proportion of generated sentences your entailment check confirms against a retrieved passage. Liu, Zhang and Liang's 51.5 percent is the number to beat, and beating it starts at knowing yours.

Appropriate decline rate: of the questions your corpus genuinely cannot answer, the share where the system said so. Sample real traffic, have a person mark which questions were unanswerable, and measure both directions. Declining too often is a failure with a cost too, and the framework for weighing the two is the one in The cost of a wrong answer: decide which error is more expensive in your situation, then set the threshold from that rather than from a default.

Neither number is owned by the model. Both are owned by whoever decides what the screen shows.

The reframe worth keeping

Treating hallucination as a model defect leads to waiting for a better model. Treating it as a design problem leads to work you can do this quarter: check each sentence against its source, show the source rather than name it, give the system a way to say no and a screen to say it on, and make disagreeing with the output a single click.

None of that reduces the rate at which the model invents things. It changes what happens next, which is the part the user experiences and the only part you can promise anything about.

The next article takes the same question into a setting where the output is not a paragraph a person reads but an action a system takes, where noticing after the fact is no longer an option.

References

1. On Faithfulness and Factuality in Abstractive Summarization. Joshua Maynez, Shashi Narayan, Bernd Bohnet and Ryan McDonald, arXiv, 2020.
2. Evaluating Verifiability in Generative Search Engines. Nelson F. Liu, Tianyi Zhang and Percy Liang, arXiv, 2023.
3. Guidelines for Human-AI Interaction: Eighteen best practices for human-centered AI design. Mihaela Vorvoreanu, Saleema Amershi and Penny Collisson, Microsoft Research, presented at CHI, 2019.
4. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez and Ion Stoica, arXiv, 2023.