Putting a model behind an interface

The provider degraded at 14:20 on a Tuesday. Requests that normally returned in four seconds started taking sixty, then failing. The application showed a spinner. After ninety seconds it showed a red toast reading Something went wrong. Please try again.

Users tried again. Each retry added load to an already struggling service. Support received forty tickets in an hour, most of them saying the product was broken, several saying their work had been lost.

Nothing in the model changed and nothing in the model would have helped. The interface had exactly two states, working and broken, for a system that has at least six.

The response times you are designing against

Nielsen's three limits are from 1993 and they have not moved, because they are about people rather than about computers. 0.1 second is about the limit for having the user feel that the system is reacting instantaneously. 1.0 second is about the limit for the user's flow of thought to stay uninterrupted, even though the user will notice the delay. 10 seconds is about the limit for keeping the user's attention focused on the dialogue.

A model call clears none of them. A short answer takes two to five seconds. A long one takes fifteen. An agent doing several tool calls, of the kind described in Agents, tools, and the limits of autonomy, takes a minute and sometimes several.

So the design question is not how to get under a second. It is what the person sees during the interval, and the honest answer is that this is where most of the engineering effort in a production system goes.

Streaming changes which number matters

The total is one number. Time to first token is another, and it is the one the user experiences as responsiveness.

Work an example. An answer of 400 output tokens at 40 tokens per second takes 10 seconds to finish. If the first token arrives at 600 milliseconds, the user is reading within Nielsen's second limit while the request continues for another ten. Without streaming, the same request is a blank screen for 10.6 seconds, which is past his third limit, the one about keeping attention on the dialogue at all.

Same latency, same cost, completely different experience. That is the reason streaming is not a nicety in these products.

It brings two obligations most implementations skip. A stream that dies mid-sentence must be visibly distinguishable from one that finished, because a truncated answer reads exactly like a complete one and the user has no way to know. And the user needs a stop control, which brings its own requirement: stopping must actually cancel the upstream request, not merely hide the output, or you keep paying for tokens nobody will read.

Progress, when there is real progress to show

Sherwin's guidance on indicators is worth applying literally. Looped animations belong on delays of roughly two to ten seconds. Percent-done indicators belong on longer processes of ten seconds or more, because they show current progress, how much has been accomplished, and how much remains, and because users grow impatient with a spinner when they cannot see themselves making progress. She notes that a percent-done indicator can suit shorter delays too when the work is a series of records, since the user understands the system is working through them.

That last case is exactly an agent, and exactly a retrieval pipeline. You usually know the steps. Searching the knowledge base, then reading four documents, then drafting. Showing them is not decoration; it is a real progress indicator over a known sequence, and it also delivers the transparency argued for in Hallucination is a design problem, because the user sees what the answer was built from before the answer arrives.

Where there genuinely is no progress information, use a looped animation and say what is happening in words. Never invent a percentage. A progress bar that lies is worse than a spinner that admits ignorance.

The six failures, and what each should say

Something went wrong covers all of these and helps with none of them.

The provider is slow. Nothing has failed. Say so: still working, this is taking longer than usual, with the stop control available. The worst response is a timeout that discards work already done.

Rate limited. The service is fine and you are asking too fast. This is the one case where automatic retry is clearly correct, and it is the case where naive retry does the most damage.

Context too long. The conversation or the retrieved passages exceed the limit. The user can act on this if you tell them, by starting a new thread or narrowing the question. They cannot act on Something went wrong.

Output truncated at the token limit. The response stopped because it hit a cap, not because it finished. Detect it from the stop reason returned by the interface, mark the answer as incomplete in the interface, and offer to continue. This one silently produces wrong answers more often than any other, because a truncated list looks like a short list.

Content filtered or refused. The system declined. This is a legitimate outcome, not an error, and it should be presented as the deliberate decision it is rather than as a fault. Retrying it automatically is the wrong behaviour in every case.

The output did not parse. You asked for structured output and got prose, or nearly-valid JSON. Retry once with the error attached, then fail into a defined state. Never show the user a parse error; they cannot fix it and it exposes your internals.

Each of those wants different copy, a different affordance, and a different retry policy. Writing them down is an afternoon's work and it is the difference between forty tickets and none.

Retry, done properly

The incident at the top of this article was made worse by users retrying, and it would have been made worse still by a client retrying automatically on a fixed timer.

Brooker's analysis of the problem is the standard reference for why. Backing off is not enough on its own, because without randomness calls cluster at predictable intervals even under exponential backoff, so the fix is not to remove backoff but to add jitter. In his simulation with a hundred contending clients, adding jitter cut the total number of calls by more than half against unjittered exponential backoff and improved time to completion substantially, spreading calls to an approximately constant rate rather than into contention spikes.

Three rules follow, and the third is the one that gets skipped.

• Exponential backoff with jitter, never a fixed interval, and a hard cap on attempts.
• Retry only what is safe to repeat. A completion is usually idempotent. A completion that already triggered a refund is not, which is where the reasoning from the previous article about placing the boundary at the irreversible step applies directly.
• Do not retry a decision. A refusal, a content filter, and a context-length error will all produce exactly the same result on the second attempt while costing you twice.

The controls the user needs

Vorvoreanu, Amershi and Collisson's list is short and most of it is unimplemented in most products. Four of the eighteen apply here almost verbatim.

Make clear what the system can do, first on their list, means the empty state does real work. A blank input box with a blinking cursor communicates that anything is possible, which is never true, and it produces the questions your system handles worst.

Convey the consequences of user actions, their sixteenth, matters most where the model writes something. Before an email is sent or a record is changed, the interface should state what will happen in the world, not summarise what the model decided.

Provide global controls, their seventeenth, is the stop button, and it is the single most requested control in every product of this kind.

Support efficient correction, their ninth, closes the loop. The user who spots a wrong answer is your best evaluator, and the report they file becomes a row in the regression set described in Evaluating a system that can say anything.

The costs that show up in the invoice

Two arithmetic points, because both are usually discovered late.

Input tokens are charged per request, so a long system prompt repeated across high volume is the dominant cost in many products. That calculation, and what to do about it, is the whole first half of Fine-tuning against prompting, a real comparison.

Output tokens are charged too, and they are also your latency. An answer twice as long takes twice as long to stream and costs twice as much, and length is rarely a property anyone measures. Asking for concision is a performance change as much as an editorial one, and it is worth measuring the average output length of your system the way you would measure the size of a page.

Track cost per request and latency per request together with the quality properties from the evaluation article, on one dashboard, and track them as distributions rather than averages. A mean latency of four seconds is consistent with a tenth of your users waiting twenty, and the warning in Train, test, and the lie of a single score about what a single number conceals applies to operational metrics exactly as it does to accuracy. Report the median and the ninety-fifth percentile side by side. A change that improves an answer while doubling the wait is a trade someone should make deliberately rather than discover from a bill.

Where this series ends up

Eight articles, and the shape of the argument has been the same each time. The model predicts tokens. The embeddings encode company rather than meaning. Retrieval bounds what can be answered. Fine-tuning changes behaviour and not knowledge. Evaluation has to be built out of properties you can actually check. Hallucination is answered on the screen. Autonomy is a per-action decision.

The interface is where all of that becomes something a person experiences. A model with unremarkable accuracy behind a well-designed interface, with honest states, real progress, easy correction and a working stop button, is a product people trust and use. An excellent model behind a spinner and a red toast is forty support tickets in an hour.

References

1. Response Times: The 3 Important Limits. Jakob Nielsen, Nielsen Norman Group, 1993.
2. Progress Indicators Make a Slow System Less Insufferable. Katie Sherwin, Nielsen Norman Group, 2014.
3. Exponential Backoff And Jitter. Marc Brooker, Amazon Web Services Architecture Blog, 2015.
4. Guidelines for Human-AI Interaction: Eighteen best practices for human-centered AI design. Mihaela Vorvoreanu, Saleema Amershi and Penny Collisson, Microsoft Research, presented at CHI, 2019.