Cost per Solved Task: Break-Even Pass Rates, pass@k and Routing

The number that quietly moved

A team reports good news: a new model for its coding agent is twice as fast and costs about half as much per attempt. Nobody changed the review process. A few weeks later the reviewers are behind and nobody can say why, because the dashboard still shows the generation line going down. A cost improvement in one stage can raise the cost of the result, and you only see it by pricing solved tasks instead of attempts.

This is a routine applied-AI decision: which model, how many samples, and whether to route between them. It is also a common interview exercise, because it tests whether you price the whole pipeline or only the metered part.

Derive cost per solved task

A review queue is any pipeline where a model produces a candidate, a check or a person evaluates it, and only accepted candidates count. Let g be the generation cost per attempt, r the review cost per attempt (every attempt is reviewed, including those that fail), and p the pass rate. Over N attempts the cost is N·(g + r) and the number solved is N·p, so

The same formula falls out of retries. If a failed attempt is detected and retried independently, the number of attempts until success is geometric with mean 1/p, so the expected cost of one solved task is (g + r)/p. If nothing passes, the ratio is undefined: report the spend and zero solved, not a cost of zero.

Work the sensitivity table

A constructed teaching scenario in one abstract cost unit, not a result from any deployment. Baseline: generation 0.10 and review 0.50 per attempt, and 80 of 100 attempts pass.

A new model halves generation cost, and the pass rate falls to 40%.

Cost per solved task rose from 0.75 to 1.375, about 83%, while the generation line shows a 50% saving. With generation and review per attempt fixed, the pass rate decides it:

The break-even pass rate for the new model sets its cost per solved task equal to the old one: 0.55/p = 0.75, so p = 0.55/0.75 ≈ 0.733. In a batch of 100 you need at least 74 solved: 55/73 ≈ 0.753 is slightly worse than baseline, and 55/74 ≈ 0.743 slightly better.

View full-size diagram

View full-size diagram

Review dominates here, so generation price barely moves the numerator while the pass rate moves the denominator. When review is automated and cheap, the balance shifts toward generation cost. What a Claude Code task costs derives g for a real agent loop from token counts and dated prices.

Put an error bar on the pass rate

The pass rate is an estimate, and cost per solved task inherits its uncertainty. From 100 attempts at 40%, a binomial 95% interval is 0.4 ± 1.96·√(0.4 × 0.6 / 100) = 0.4 ± 0.096. So cost per solved task lies roughly between 0.55/0.496 ≈ 1.11 and 0.55/0.304 ≈ 1.81. Both ends are above the baseline's 0.75, so this comparison survives its uncertainty. A pass rate near the 73% break-even would not: you would need more attempts before switching. Chapter 16 of the book derives the interval and shows how to compare two models on the same tasks, which needs fewer cases than comparing them on different ones.

pass@k: more samples, if the check is cheap

Drawing k samples per task and keeping any that passes raises the solve rate. If samples were independent with pass rate p, the chance at least one passes is

At p = 0.4, pass@3 = 1 − 0.6³ = 0.784. That looks like a way back to the baseline's 80%, but it assumes two things. First, a cheap automatic verifier, such as a test suite, picks the passing sample. If a person reviews every sample, review cost is paid k times: 3 × (0.05 + 0.50) = 1.65 per task, and 1.65/0.784 ≈ 2.10 per solved task. Second, independence. Samples on the same task are correlated: a task the model cannot do, it tends to fail every time. Estimate pass@k from repeated samples per task, not from the formula; Chapter 16 explains the unbiased estimator and why the obvious one is wrong.

Routing: a cascade between two models

A cheap model A and a strong model B. A constructed example with an automatic test as the verifier:

A looks three times cheaper per solved task, but only if its retries are independent, which the last section says they are not. A cascade is more honest: try A once, and send failures to B.

Here q is B's pass rate on the tasks A failed, which is lower than B's overall 0.8 if hard tasks are hard for both. At q = 0.6, the solve rate is 0.8 and cost per solved task is 0.73/0.8 ≈ 0.91, cheaper than B alone at the same solve rate. At q = 0.3 the solve rate falls to 0.65. The decision turns on q, and q can only be measured by running B on A's failures.

Check the decision

The review queue that limits throughput

A lower pass rate also spends reviewer time on work that gets rejected, which lowers how many solved tasks a fixed review capacity produces, apart from the cost arithmetic. And a mean review time can hide a tail. Two constructed lists both average 4 minutes:

cases_a = [4, 4, 4, 4, 4]
cases_b = [1, 1, 1, 1, 16]

def mean(xs):
    return sum(xs) / len(xs)

def worst_case_violates(xs, limit):
    return max(xs) > limit

print(mean(cases_a), mean(cases_b))       # 4.0 4.0
print(worst_case_violates(cases_a, 10))   # False
print(worst_case_violates(cases_b, 10))   # True

If the team escalates any review over 10 minutes, only the second list breaks the rule, though the means match. Five invented values are not a p95; the latency article derives how many observations a tail percentile needs.

What an interviewer would ask

Expect a price table and two pass rates, and a question like: which model would you deploy, and what would change your mind? A strong answer computes cost per solved task including review, finds the break-even pass rate, puts an interval on each pass rate, asks whether a verifier exists before proposing pass@k, and asks for the conditional pass rate before proposing a cascade. In a lab the same arithmetic prices an evaluation budget, a data-generation pipeline with a filter, or best-of-k sampling with a reward model.

What would change this conclusion

If review cost differs by outcome, for example a quick triage for failing candidates, the numerator changes and the break-even shifts. If the acceptance gate is looser for one model, the comparison measures different things. If review time grows with candidate length, review becomes a function of output size. Each needs the table rebuilt with its assumption stated; none changes the method: price both stages, treat the pass rate as its own measured variable with an error bar, and recompute the ratio.