The reliability math of AI agents: why 95% per step is not good enough

An agent demo usually works. You watch it plan, call a few tools and land on the right answer, and it feels ready. Then it meets real users and real data, and it fails often enough that nobody trusts it.

Most of the gap is not the model being "dumb". It is arithmetic. An agent is a chain of steps, and a chain is only as reliable as the product of its links.

The compounding rule

If each step of a task succeeds independently with probability p, a task of n steps succeeds with probability pⁿ. That simple model ignores retries and correlations between steps, but it is close enough to explain why long agent runs disappoint.

xychart-beta
  title "Task success vs number of steps"
  x-axis "Steps" [1, 5, 10, 20, 30, 50]
  y-axis "Task success (%)" 0 --> 100
  line [95, 77.4, 59.9, 35.8, 21.5, 7.7]
  line [90, 59, 34.9, 12.2, 4.2, 0.5]
  line [99, 95.1, 90.4, 81.8, 74, 60.5]

Blue: 95% per-step success, the example in the text. Grey: 90%. Yellow: 99%.

Two numbers from that table are worth remembering:

• At 95% per step, a 20-step task succeeds 35.8% of the time. Most people would call a 95%-accurate component good. Chained 20 times, it fails about two runs in three.
• The point where a task becomes a coin flip comes early. At 90% per step it is about 7 steps, at 95% about 13, and even at 99% about 69.

A research or support agent can easily take 20 steps: search, read, refine the query, search again, call an API, check a policy, draft, revise. The arithmetic says it has to be extremely good at each one.

What the benchmarks show

This is not only theory. τ-bench (tau-bench) tests agents that must use tools and follow policies while talking to a simulated user, in retail and airline customer-service domains. The authors report that even state-of-the-art function-calling agents "like gpt-4o succeed on <50% of the tasks", and that consistency is worse still: "pass^8 <25% in retail".

pass^k is the important idea there. It is the probability that an agent succeeds on the same task in all of k attempts. pass^1 measures capability: can it do the task at all? pass^8 measures reliability: will it do it every time? The gap between the two is the gap between a demo and a product.

Five levers that move reliability

You can raise p, lower n, or make failures recoverable. Every effective technique does one of the three.

1. Take steps away

The cheapest step is the one you remove. If the path through a task is known, encode it as a workflow instead of letting the model decide every move. Anthropic's guidance on building agents makes the same point: "we recommend finding the simplest solution possible, and only increasing complexity when needed", and it separates workflows, where "LLMs and tools are orchestrated through predefined code paths", from agents that direct their own process.

Cutting a task from 20 steps to 8 at 95% per step lifts success from 35.8% to 66.3%, with no change to the model.

2. Raise per-step accuracy where it is cheapest

The steps that fail most in practice are often retrieval steps: the agent searches, gets the wrong context, and every later step builds on it. Better retrieval raises p and lowers n at the same time, because an agent that finds the right document first time does not need to search again.

The BrowseComp-Plus benchmark shows this cleanly. With the same GPT-5 agent on the same 100,195-document corpus, swapping a BM25 retriever for Qwen3-Embedding-8B raised accuracy from 55.9% to 70.1% and reduced the average number of searches per question from 23.2 to 21.7. We cover this in depth in Your agent is only as good as its retriever.

3. Design tools like interfaces

Every tool call is a step that can go wrong: the wrong tool, the wrong arguments, a misread response. Anthropic recommends investing "just as much effort in creating good agent-computer interfaces (ACI)" as in human interfaces. In practice:

• Few, distinct tools. Two overlapping search tools invite the wrong choice.
• Narrow, typed arguments. Prefer category: "billing" | "shipping" to free text.
• Errors that explain. "No orders found for customer 4821 in the last 90 days" lets the agent recover; 500 Internal Server Error does not.
• Results sized for the context. Return the five relevant fields, not the whole record.

4. Verify and retry

A check after a step turns a silent failure into a retry. As a simple model, if a check catches a fraction c of failed steps and the retry succeeds with the same probability p, the effective per-step success becomes p + (1 − p) · c · p.

Computed from the formula above with p = 95%; real checks and retries are not independent, so treat this as a direction, not a forecast.

Good checks are cheap and specific: does the SQL parse, did the API return the expected schema, does the answer cite a retrieved document, does the order total match the line items.

5. Put a human in front of irreversible actions

Some steps should never run on probability alone. Refunds, deletions, outbound emails and production changes deserve an approval step and narrowly scoped credentials. This does not raise p; it makes the worst failures impossible.

flowchart LR
  P[Plan step] --> V{Arguments valid?}
  V -- no --> P
  V -- yes --> R{Irreversible?}
  R -- yes --> H[Human approval]
  R -- no --> C[Call tool]
  H --> C
  C --> K{Result checks pass?}
  K -- yes --> N[Next step]
  K -- no, retries left --> P
  K -- no, out of retries --> E[Escalate to a person]

Measure reliability, not just capability

If you only run each eval task once, you are measuring pass^1 and learning what the agent can do. Run each task several times and report how often it succeeds on all of them. A useful minimum:

1. A fixed task set drawn from real traffic, with clear pass/fail criteria per task.
2. Several runs per task so you can report pass^k alongside pass^1.
3. Per-step logging so a failure points to the step that caused it: retrieval, tool choice, arguments or reasoning.
4. A regression gate that blocks a prompt, model or tool change if pass^k drops.

Before an agent touches production

• Count the steps in a typical task and compute pⁿ for your measured per-step success. If the result is not acceptable, the design has to change before the model does.
• Replace known paths with workflows; keep agent autonomy for the genuinely open-ended parts.
• Audit retrieval quality first. It is usually the cheapest step to improve.
• Give every tool typed arguments, explanatory errors and compact results.
• Add a check after every step that has a cheap, specific test.
• Require human approval and scoped credentials for anything irreversible.
• Report pass^k, not just pass^1, and gate releases on it.

Sources

• Yao et al., τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, arXiv 2406.12045.
• Anthropic, Building effective agents, 19 December 2024.
• Chen et al., BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent, arXiv 2508.06600 (Table 1).