Reliability Compounds: The Demo Worked, the Task Failed

A demo runs an agent for a few steps on an input someone prepared. A real task runs it for many steps on whatever arrives. The same model can look solid in the first setting and fail most of the time in the second, and you do not need a theory of intelligence to explain why. You need one line of probability, and then the discipline to check where that line's assumptions break.

Derive it: reliability over n steps

Model an agent task as n steps. Each step succeeds with probability p, the steps fail independently, and the task succeeds only if every step does. Then:

P(task) = pⁿ

Take logs to see how fast it falls. For p close to 1, ln p ≈ −(1 − p), so

P(task) ≈ e^(−n(1 − p))

The task success halves every ln 2 / (−ln p) steps. At p = 0.95 that is 0.693 / 0.0513 ≈ 13.5 steps. At p = 0.99 it is about 69 steps. Per-step failure rate, not per-step accuracy, sets the scale.

Run it backward and you get a requirement. For a 20-step task to succeed 90% of the time, each step needs p = 0.9^(1/20) ≈ 0.9947. That is roughly one failure in 190 steps. A three-step demo at p = 0.95 succeeds 86% of the time, which is why it looks fine.

Chapter 3 of the book builds the loop this formula describes and derives the same result from its own traces.

Put an error bar on p

You never know p. You estimate it. Suppose you run one tool-calling step 200 times and 190 succeed. The estimate is p̂ = 0.95, and the 95% binomial interval is

1.96 · √(0.95 · 0.05 / 200) = 1.96 · 0.0154 ≈ 0.030

so p is somewhere in [0.920, 0.980]. Push both ends through ten steps: 0.920¹⁰ ≈ 0.43 and 0.980¹⁰ ≈ 0.82. A ±3-point interval on one step becomes a 0.43-to-0.82 interval on the task. The exponent amplifies your uncertainty along with your failure rate.

That is the first habit this course asks for: never report an agent success rate without n and an interval. Chapter 16 covers the interval choices, including when the simple formula above misbehaves near 0 or 1.

Where independence breaks: pass@k and pass^k

The formula assumes independent steps. Real failures are not independent: some tasks are hard for every run, and some are easy for every run. Two metrics separate these cases.

• pass@k is the probability that at least one of k attempts succeeds. It fits work where one success is enough and you can check it, such as generating candidate solutions against a test suite.
• pass^k is the probability that all k attempts succeed. It fits work where consistency is the product, such as an agent that serves the same kind of request thousands of times.

Anthropic's engineering post Demystifying evals for AI agents (January 2026) defines both this way. The pass^k metric was introduced by the τ-bench paper (Yao et al., 2024), which reports that even strong function-calling agents (like gpt-4o) "succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)."

If every attempt had the same success probability s, independently, you would predict pass@k = 1 − (1 − s)ᵏ and pass^k = sᵏ. At s = 0.5 and k = 8 that gives pass^8 = 0.0039. Measured pass^k is usually higher than that, and the reason is informative. Suppose half your tasks succeed with probability 0.95 and half with 0.05. Average success is still 0.5, but

pass^8 = 0.5 · 0.95⁸ + 0.5 · 0.05⁸ ≈ 0.332

which is 85 times the independence prediction. Because xᵏ is convex, averaging it over tasks of mixed difficulty always gives at least the k-th power of the average (Jensen's inequality). A measured pass^k far above sᵏ tells you failures cluster on specific tasks. That is where to read traces first.

What a verifier plus a retry does

Retries are the obvious fix for per-step failure, and they only work on failures you can see. Split each attempt's outcome three ways:

• success, with probability p;
• a detected failure, probability d: a schema check, a test or a verifier catches it;
• a silent failure, probability s: the step returns something wrong that nothing flags.

With p + d + s = 1 and up to r attempts, retrying only on detected failures, the step succeeds with probability

p · (1 + d + d² + … + d^(r−1)) = p · (1 − dʳ) / (1 − d)

As r grows this approaches p / (1 − d) = p / (p + s). Silent failures set the ceiling.

View full-size diagram

Two things follow. The second attempt buys almost everything; after that you are paying for tokens, not reliability. And the only way past the ceiling is to move failures from silent to detected. That is the job of an evidence gate, deterministic code that refuses a step's output unless the evidence it names is present. Lessons 04 and 05 build one, and the course's offline evidence-gate lab lets you run one in a minute.

The demo and the task are different experiments

A demo is evidence that a task can succeed. It is not an estimate of how often it does. The two differ by the exponent, by the input distribution and by the checks around each step.

What this means in a lab

When an applied-AI or research team sees an agent fail, the first questions are arithmetic ones. How many steps? What is the per-step failure rate, measured on how many runs? Which failures are detected, and which are silent? Is the reported number pass@k or pass^k? You can answer each of those from traces, without an opinion about the model. The model may still be the limit. You find that out after the arithmetic, not instead of it.

To practice, the book's course repo has a reliability and retries exercise for Chapter 3 that runs on Colab.