Model your agent as a chain of steps and see how per-step success compounds into end-to-end success, and what retries, checks and fewer steps buy you in reliability, cost and latency. The companion to the reliability math of AI agents.
Illustrative values, not benchmarks. Replace them with your own measurements.
Effective success 95.0%, 1.00 expected attempts
Effective success 95.0%, 1.00 expected attempts
Effective success 95.0%, 1.00 expected attempts
Effective success 95.0%, 1.00 expected attempts
Effective success 95.0%, 1.00 expected attempts
Effective success 95.0%, 1.00 expected attempts
Effective success 95.0%, 1.00 expected attempts
Effective success 95.0%, 1.00 expected attempts
Enter your model's prices to see cost. Left at 0, cost is not shown.
66.3% without retries or checks, across 8 steps.
Enter your model's prices to see cost.
Plan the research
We design and build agents for production reliability: see AI agent development.
End-to-end success. With independent steps, P = ∏ pi', the product of each step's effective success. At 95% per step, 8 steps give 0.958 = 66.3% and 20 steps give 35.8%, as in the reliability math of AI agents.
Retries. A retry only runs if the failure is noticed. If failures are always detectable (a tool error, a schema check) and a step allows r retries, p' = 1 − (1 − p)r+1, and expected attempts = Σk=0..r (1 − p)k. Silent failures get no retry, so p' = p.
Verifier (an assumption, stated plainly). We model a verifier as catching a fixed fraction c of failed attempts, independently of the attempt, and each retry succeeding with the same p. A failed attempt is retried with probability q = (1 − p)·c, so expected attempts = Σk=0..r qk and p' = p · Σk=0..r qk. With one retry this is the post's p + (1 − p)·c·p (c = 80% lifts 95% to 98.8% per step). It never flags a correct result, and its own tokens and latency are not counted; if it is an LLM call, add it as a step. With 0 retries a verifier changes nothing here (in practice it escalates).
Human approval does not raise p in this model. It adds your wait time to latency and, as the post argues, makes irreversible failures preventable rather than less likely.
Cost and latency. Cost = Σ attemptsi × (tokens in × input price + tokens out × output price) / 1,000,000. Latency = Σ attemptsi × latencyi + approvals × wait, assuming steps run in sequence and every step runs (a silent failure is only discovered at the end). The biggest contributor is the step with the largest share of −ln P, which is how failures add up across a product.
This is a planning tool, not a prediction. It assumes independent failures. Real agents have correlated errors: a bad retrieval poisons every later step, and a retry often repeats the same mistake, so real retries and checks help less than shown. Measure per-step success on your own traffic and run each task several times (pass^k, from τ-bench) before trusting any number here.
Sources: The reliability math of AI agents; Yao et al., τ-bench (arXiv 2406.12045); Anthropic, Building effective agents.
Because step success multiplies. If each step succeeds independently with probability p, n steps succeed with probability p to the power n: 0.95 over 8 steps is 66.3%, and over 20 steps 35.8%.
Only as much as failures are detected. If a failure is always noticed and a step gets r retries, its success becomes 1 - (1 - p)^(r + 1): one retry lifts 95% to 99.75%. A silent failure gets no retry, so it gains nothing.
As an assumption: the verifier catches a fixed fraction of failed attempts and each retry succeeds with the same probability. With one retry, effective success is p + (1 - p) x c x p, so a check that catches 80% of failures lifts 95% to 98.8% per step.
No. The model assumes independent failures, but real agents have correlated errors: one bad retrieval affects every later step, and retries often repeat a mistake. Use it to compare designs, then measure pass^k on your own tasks.
We map your agent's steps, measure where it fails, and redesign it with fewer steps, better retrieval and cheap checks.
