Evaluating a system that can say anything

A prompt change ships on Friday. The evaluation suite moves from 71 percent to 76 percent, the pull request is approved on the strength of that number, and the release goes out. By Tuesday, tickets complaining that the assistant gave the wrong answer have roughly doubled.

The suite was sixty questions. They were written by the engineer who wrote the prompt, in an afternoon, and graded by comparing the model's output to a stored string. The prompt change had made answers shorter. Shorter answers matched the stored strings more often. Nothing about the system had improved.

Every part of that is ordinary, and every part of it is avoidable.

Why the old machinery does not transfer

For a classifier there is one right answer per row, so accuracy, precision and recall are defined without argument, and the reasoning in The cost of a wrong answer tells you which to weight. The discipline in Train, test, and the lie of a single score tells you how to split the data so the number means something.

A generative system breaks the first assumption. There are many correct answers, differing in wording, order, length and register, and there is no way to enumerate them. So exact match understates, and any similarity metric loosens the definition of correct until it stops tracking what you care about.

The choice of metric is not a technical detail either. Schaeffer, Miranda and Koyejo examined the claim that large models display abilities absent from smaller ones, appearing sharply and unpredictably at particular scales. Their alternative explanation is that for a particular task and model family, when analysing fixed model outputs, emergent abilities appear due to the researcher's choice of metric rather than due to fundamental changes in model behaviour with scale. They state the mechanism plainly: nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous, predictable changes in performance. Alleged emergent abilities, they conclude, evaporate with different metrics or with better statistics.

Fixed outputs. Same model, same answers, different metric, different story. If a metric can manufacture a discontinuity in a research result, it can certainly manufacture a five-point improvement in your Friday release.

Cut the problem into pieces you can grade

The trick that makes evaluation tractable is refusing to grade the whole output at once.

Most production systems contain sub-tasks with genuinely correct answers, and those can be graded like any classifier. Did it pick the right category out of eleven. Did it extract the invoice number that is in the document. Did the output parse as the schema you asked for. Did the retrieval step return the passage a person marked as the answer, and at what rank, which is the separate retrieval evaluation argued for in Retrieval-augmented generation, honestly.

For the free-text part, replace correctness with a set of properties that are each checkable:

• Grounded. Every claim in the answer is supported by the retrieved passages. This is checkable against a specific text, unlike truth in general.
• Answered or declined appropriately. Given a question the corpus cannot answer, did it decline.
• Complete. Does the answer contain the facts a person marked as required. This is a recall check over a short list, not a judgement about prose.
• Compliant. No prices unless the source states one. No medical advice. Whatever your rules are, written down and checked individually.

Each of these has a yes or no answer that two people would usually agree on. That property is what makes an evaluation set worth building, and it is worth more than any composite score.

What a public benchmark is actually evidence of

Public benchmarks answer a different question from yours, and both halves of that sentence matter.

Liang, Bommasani, Lee and their colleagues built the most careful attempt at breadth. Their motivation is that language models are becoming the foundation for almost all major language technologies while their capabilities, limitations and risks are not well understood, and their response was to evaluate 30 prominent models across 16 core scenarios on 7 metrics covering accuracy, calibration, resistance to perturbed inputs, fairness, bias, toxicity and efficiency, with 7 targeted evaluations over 26 further scenarios. The number that shows why the effort was needed is the coverage improvement: before their work, prior benchmarking covered 17.9 percent of the model-by-scenario grid, and they raised it to 96.0 percent, so all 30 models were densely benchmarked on the same scenarios and metrics under standardised conditions.

Read what that implies about the state of the field at the time. Most published comparisons were between models that had not been run on the same things. A leaderboard is a claim about the specific scenarios it contains, under its own conditions, and nothing more.

There is a second problem, and it gets worse every year. Sainz, Campos, Garcia-Ferrero, Etxaniz, Lopez de Lacalle and Agirre argue that classical evaluation on annotated benchmarks is in trouble. The worst kind of data contamination, they write, happens when a model is trained on the test split of a benchmark and is then evaluated on the same benchmark, and the extent of the problem is unknown because it is not straightforward to measure. Contamination causes an overestimation of the performance of a contaminated model relative to non-contaminated counterparts.

Any benchmark published before a model was trained may be inside that model's training data. That is not an accusation against anyone; it is a property of training on a crawl of the public internet. Your own evaluation set, built from your own data, never published, does not have this problem, which is a large and underrated reason to build one.

Using a model as the grader

Grading free text by hand does not scale, so the standard move is to have a strong model grade the output. It works better than most people expect and it fails in specific, documented ways.

Zheng and colleagues measured both sides. They explore using strong models as judges on open-ended questions, and they verify agreement against human preferences using a multi-turn question set and a crowdsourced comparison platform. Their result is that strong judges such as GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80 percent agreement, the same level of agreement as between humans.

That is a genuinely useful ceiling: a model judge agrees with people about as often as people agree with each other. It is also a ceiling, and the same paper names what sits underneath it. They examine the usage and limitations of the approach, including position, verbosity and self-enhancement biases, as well as limited reasoning ability.

Take those one at a time, because each has a countermeasure. Position bias means the order in which two answers are shown changes which one wins, so run every comparison in both orders and discard pairs where the verdict flips. Verbosity bias means longer answers score higher, so control for length or measure it separately and watch whether your improvements are just words. Self-enhancement bias means a judge favours output resembling its own, so do not use the same model as generator and grader when the comparison matters.

And do not let a judge score an absolute quality out of ten. Ask it a narrow yes or no question about a specific property with the source text in front of it, which is the format the properties above were designed for.

Human review is the only ground truth, so spend it carefully

Someone has to look. The question is only how to make looking cheap enough to happen weekly.

Sample from real traffic, not from questions you invented, because invented questions are systematically the ones you already thought of. Blind the reviewer to which version produced which answer. Prefer pairwise comparison over absolute scoring: which of these two is better is a question people answer consistently, and rate this eight out of ten is not.

Measure your reviewers against each other before you trust them against the system. If two people disagree on a third of the sample, no model score derived from their labels means anything, and the disagreement is usually a sign that the rule they are applying was never written down clearly. Fixing that is the real work, and it improves the product as much as the evaluation.

The harness that pays for itself

Keep three things.

• A regression set of every failure anyone has reported, with the expected behaviour written beside it. It only grows. This catches the thing you already fixed coming back, which is the most common release defect in these systems.
• A sampled review of real traffic, thirty to fifty items a week, reviewed by a person against the properties above. This finds failures nobody thought to write a test for.
• A counter for declines, split into declines that were correct and declines that were not. A system that never declines and a system that declines constantly both look fine on an accuracy metric and neither is acceptable, which is the subject of the next article.

Report the properties separately. A single composite number invites the Friday failure at the top of this article, because it lets an improvement in one property hide a regression in another. Report groundedness, completeness, decline rate and compliance side by side, and let the reader see the trade the release made.

None of this is sophisticated. It is the ordinary discipline of measurement, applied to a system whose output happens to be prose, and it is nearly always the difference between a team that knows whether it is improving and one that argues about it.

References

1. Holistic Evaluation of Language Models. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga and colleagues, arXiv, 2022.
2. Are Emergent Abilities of Large Language Models a Mirage?. Rylan Schaeffer, Brando Miranda and Sanmi Koyejo, arXiv, 2023.
3. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez and Ion Stoica, arXiv, 2023.
4. NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark. Oscar Sainz, Jon Ander Campos, Iker Garcia-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle and Eneko Agirre, arXiv, 2023.