Agents, tools, and the limits of autonomy

An agent is given three tools: search the order database, issue a refund, and email the customer. It handles a straightforward request correctly. It handles the next one correctly. On the third, the search returns two orders for the same customer, the agent picks the wrong one, and refunds an order that was never disputed.

The refund is issued. There is no review step, because the first two worked.

That sequence contains everything difficult about agents. The capability was real. The average was good. The failure was silent, irreversible, and produced by the ordinary case of an ambiguous lookup rather than by anything exotic.

What an agent adds to a model

The loop is short enough to describe in a sentence: the model produces an action rather than an answer, something executes it, the result is appended to the context, and the model produces the next action.

Yao, Zhao, Yu, Du, Shafran, Narasimhan and Cao named the pattern and showed why interleaving matters. Reasoning and acting had been studied separately, so they explored generating both reasoning traces and task-specific actions in an interleaved manner: reasoning traces help the model induce, track and update action plans as well as handle exceptions, while actions allow it to interface with external sources such as knowledge bases or environments to gather additional information. On question answering and fact verification, they report that interacting with a simple Wikipedia interface overcomes issues of hallucination and error propagation that are common in reasoning-only prompting. On two interactive decision making benchmarks they outperform imitation and reinforcement learning methods by an absolute success rate of 34 percent and 10 percent respectively, while being prompted with only one or two in-context examples.

The mechanism worth carrying forward is the first half. Acting fixes a category of error that reasoning alone cannot, because a step that consults the world can be corrected by the world. This is the same argument as Retrieval-augmented generation, honestly, extended from reading to doing.

It also introduces the failure this article is about. The model now writes to systems, and a wrong action is not a sentence a reader can discount.

The average is the wrong statistic

Most agent claims are averages over a task set. Averages hide the property that actually determines whether you can deploy something.

Yao, Shinn, Razavi and Narasimhan built a benchmark to test the thing production systems need, noting that existing benchmarks do not test agents on their interaction with human users or their ability to follow domain-specific rules. Theirs emulates dynamic conversations between a simulated user and an agent equipped with domain-specific tools and policy guidelines, and evaluates by comparing the database state at the end of a conversation with an annotated goal state, which is a far harder standard than judging a transcript.

Their headline result is that even state-of-the-art function calling agents succeed on under 50 percent of tasks. The result that matters more is the second one. They propose a metric for the reliability of agent behaviour over multiple trials, and report that agents are quite inconsistent, with pass to the eighth under 25 percent in the retail domain.

Sit with the difference between those two numbers. Under half of tasks solved on a single attempt, and under a quarter of tasks solved on all of eight attempts. The agent that handled your refund correctly this morning is meaningfully likely to handle the identical case differently this afternoon. Their own conclusion is that this points to the need for methods that improve the ability of agents to act consistently and follow rules reliably.

Any process design that assumes stability across runs is designing against measured behaviour. Sampling makes variation the default, as Language models predict the next token set out, and a loop of ten sampled steps compounds it.

Long-horizon work is much harder than short demos suggest

The other systematic overestimate comes from watching a five-step task and extrapolating.

Jimenez, Yang, Wettig, Yao, Pei, Press and Narasimhan built an evaluation from real software engineering work: 2,294 problems drawn from real GitHub issues and their corresponding pull requests across 12 popular Python repositories, where the model is given a codebase and an issue description and must edit the codebase to resolve it. They describe why it is hard, and the description generalises past code: resolving these issues frequently requires understanding and coordinating changes across multiple functions, classes and even files simultaneously, calling for models to interact with execution environments, process extremely long contexts, and perform complex reasoning that goes beyond traditional code generation.

Their measurement at the time was that both state-of-the-art proprietary models and their own fine-tuned model could resolve only the simplest issues, with the best performing model solving a mere 1.96 percent of the issues.

That number is from 2023 and it has moved a great deal since, so do not carry the figure. Carry the shape of the finding: performance on tasks requiring coordinated changes across a large state was an order of magnitude below performance on tasks that fit in one step. Whatever the current numbers, check them for the horizon your work actually has rather than the horizon of the demo.

The failure modes that are specific to agents

The irreversible action taken on an ambiguous read. The refund at the top of this article. The lookup returned two rows, the model chose one, and nothing in the loop distinguished a confident selection from a coin flip. Ambiguity in a tool result is where a human belongs, and it is detectable in code: if the query returned more than one match, do not let the model pick silently.

Compounding error across steps. A ten-step task where each step is right nine times in ten completes about 35 percent of the time. Not because any step is unreliable, but because 0.9 to the tenth is 0.349. Agents fail at length for arithmetic reasons before any interesting reason.

Poisoned context from a tool result. This is the one most teams have not thought about. Greshake, Abdelnabi, Mishra, Endres, Holz and Fritz showed that applications integrating these models blur the line between data and instructions, and revealed attack vectors using indirect prompt injection that let adversaries remotely exploit such applications, without a direct interface, by strategically injecting prompts into data likely to be retrieved. They derive a taxonomy covering data theft, worming, information ecosystem contamination and other risks, and demonstrated practical attacks against real-world systems including a major search chat product and code-completion engines. They show how processing retrieved prompts can act as arbitrary code execution, manipulate the application's functionality, and control how and if other tools are called. Their assessment at the time was that effective mitigations were lacking.

The practical version: any text your agent reads can attempt to instruct it. A web page, a support email, a PDF in a shared drive, a code comment. If your agent reads customer-supplied text and can also send email or write to a database, someone can eventually make it do so. Treat every tool result as untrusted input, and put the authorisation check in the tool rather than in the prompt, because a prompt instruction is a request and a code check is a constraint.

Loops that do not terminate. Retry, refine, try again, refine. Cap steps, cap cost, cap wall-clock time, and make the cap a visible outcome rather than a crash.

Confident completion reports. The agent says the refund was processed. It was not; the call returned an error the model summarised optimistically. Never take the model's word for what happened. Read the tool's return value in code and derive the status from that.

Where supervision belongs

Not everywhere. A review step on every action produces alert fatigue within a week, and a human who approves everything is worse than no human because the process now claims oversight it does not have.

Place the human by the properties of the action, decided in advance and enforced in code.

• Reversibility. Drafting an email is reversible. Sending it is not. Put the boundary at the irreversible step, not at the start of the task.
• Blast radius. One record or ten thousand. A bulk operation deserves review even when the single-record version does not.
• Ambiguity in the inputs. More than one match, a low retrieval score, a missing field. These are computable conditions, so make them route to a person automatically.
• Cost of the error against cost of the check. The framework in The cost of a wrong answer applies directly: an approval step has a real price in delay and attention, and it is worth paying only where the error it catches costs more.

Then make the review meaningful. Show what the agent will do, the tool result it is acting on, and the alternative it rejected. An approval dialogue that shows only a summary sentence is the interface failure described in Hallucination is a design problem, moved somewhere with worse consequences.

Building one you can actually run

• Give the fewest tools that cover the task. Each additional tool multiplies the ways a plan can go wrong and lengthens the prompt.
• Make every tool validate its own arguments and check its own authorisation. Assume the model will call it with wrong values, because it will.
• Log every step: the action, the arguments, the raw result. When something goes wrong, the transcript is the only evidence, and it is also your evaluation set.
• Test the same task repeatedly, not once. Report the proportion of runs that succeed, not whether a run succeeded. That is what the reliability metric above exists to express, and it is the number a business needs.
• Make cost and step count first-class limits, visible to whoever runs the thing.

Autonomy is not a property a system has or lacks. It is a decision about which specific actions run without a person, taken action by action, on evidence about how often that action goes wrong and what it costs when it does.

The last article in this series is about what the person on the other side of all this actually sees: how long they wait, what appears while they wait, and what the screen says when the whole thing fails.

References

1. ReAct: Synergizing Reasoning and Acting in Language Models. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan and Yuan Cao, arXiv, 2022.
2. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz and Mario Fritz, arXiv, 2023.
3. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press and Karthik Narasimhan, arXiv, 2023.
4. tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan, arXiv, 2024.