Route Between Models by Cost per Accepted Outcome

A routing question is a measurement question

You run an agent that reads papers and technical reports and fills rows in your team's comparison table: benchmark, metric, value, and the table or figure the value came from. A large model does the job well. A small model costs about a tenth as much per call. Someone asks whether most requests can go to the small one.

A leaderboard cannot answer that. It reports an average over someone else's tasks, graded by someone else's rules. What you need is narrower: on your own cases, graded by your own acceptance rule, how much does each configuration cost per row you would actually keep?

That quantity has a short derivation, and it changes the answer more often than people expect. This article derives it, puts error bars on it, and turns it into a routing study you can run on your own agent.

Derive the cost per accepted outcome

Let one attempt cost c: input tokens times the input price, plus output tokens times the output price. Let your grader accept each attempt with probability p. If a rejected attempt is caught and retried, and attempts are independent, the number of attempts until the first accepted one is geometric with mean 1/p. So:

expected cost per accepted outcome = c / p

Most pipelines also pay to check each attempt: a grader model, a script, or a person. Call that cost g per attempt:

expected cost per accepted outcome = (c + g) / p

Two configurations, large L and small S, cost the same per accepted outcome when (c_S + g) / p_S = (c_L + g) / p_L. Solve for g and you get the crossover:

g* = (p_S * c_L - p_L * c_S) / (p_L - p_S)

Below g*, the cheap model wins despite its lower acceptance rate. Above it, the accurate model wins, because every rejected attempt costs you another check.

One assumption needs a warning label. Retrying assumes attempts are independent. Chapter 16 of the book found the opposite in its retained run: every case gave the same result on both of its repeats, so a case that failed once failed again. For hard cases, c / p is optimistic. Measure the retry acceptance rate separately before you rely on it.

A worked example with the arithmetic shown

The numbers below are a worked example, not a measurement. The prices are illustrative, not a quote for any model, and the counts are invented so you can check the arithmetic. The protocol later in the article replaces them with yours.

Each extraction request sends about 6,000 input tokens and returns about 400 output tokens.

Freeze the task families and an acceptance rule for each before either configuration runs:

Suppose the graded runs come back like this:

Take the reported-metric family, where both models are plausible. With no checking cost:

• L: 0.024 / (134/140) = 0.024 / 0.957 ≈ $0.0251 per accepted row.
• S: 0.002 / (124/140) = 0.002 / 0.886 ≈ $0.0023 per accepted row.

The small model is about eleven times cheaper per kept row. Now add a checking cost. The crossover is:

g* = (0.886 × 0.024 − 0.957 × 0.002) / (0.957 − 0.886)
   ≈ 0.0193 / 0.071
   ≈ $0.27 per attempt

If a person spends a minute confirming each row against the paper, and that minute costs more than about 27 cents, the large model is cheaper per accepted row. At g = $0.50, L costs 0.524 / 0.957 ≈ $0.548 and S costs 0.502 / 0.886 ≈ $0.567. When verification dominates, acceptance rate dominates.

Put error bars on both terms

An acceptance rate of 124 out of 140 is an estimate. Chapter 16 derives the Wilson interval and shows why the textbook interval, p ± 1.96·√(p(1−p)/n), fails near 0 and 1. For these counts, the 95% Wilson intervals are:

Because c is fixed, the cost range runs from c / p_high to c / p_low. The two ranges are far apart, so the no-checking decision is stable. With g = $0.50 the point estimates sit two cents apart, while the ranges are several cents wide: L from about $0.53 to $0.58 per accepted row, S from about $0.54 to $0.61. That comparison is inside the noise, and an honest report says so.

The same cases were graded under both configurations, so compare them pair by pair. Of the 140 reported-metric cases, suppose 120 passed under both, 14 only under L, 4 only under S, and 2 under neither. Only the 18 discordant cases carry evidence about which is better. McNemar's exact test, also derived in Chapter 16, gives p ≈ 0.031 for 14 against 4. The paired comparison can see a difference that two separate scores with overlapping intervals would blur.

Here is the arithmetic as a script. It runs on Colab or any Python 3.12, standard library only:

import math


def wilson(k, n, z=1.96):
    p = k / n
    d = 1 + z * z / n
    center = (p + z * z / (2 * n)) / d
    half = z / d * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return center - half, center + half


def cost_per_accepted(c, g, k, n):
    low, high = wilson(k, n)
    return (c + g) / (k / n), (c + g) / high, (c + g) / low


def mcnemar_exact(b, c):
    tail = sum(math.comb(b + c, i) for i in range(min(b, c) + 1))
    return min(1.0, 2 * tail / 2 ** (b + c))


for g in (0.0, 0.50):
    for name, c, k in (("large", 0.024, 134), ("small", 0.002, 124)):
        point, best, worst = cost_per_accepted(c, g, k, 140)
        print(f"g={g:.2f} {name}: ${point:.4f} (range ${best:.4f} to ${worst:.4f})")
print("McNemar p, 14 vs 4 discordant:", round(mcnemar_exact(14, 4), 4))

It prints the four cost ranges above and McNemar p, 14 vs 4 discordant: 0.0309. Swap in your own counts and prices.

View full-size diagram

Route by family, not by average

The aggregate row, 188 against 142 out of 200, hides the decision. Read the families.

Missing metric. S stopped correctly in 9 of 30 cases. In the other 21 it wrote a number the paper does not contain. A fabricated value in a results table is worse than an empty cell, because it looks exactly like a real one. No price difference buys that back. L passed 29 of 30, with a Wilson interval of 0.833 to 0.994: good, and still not proof that it never fabricates.

Held-out split. Neither configuration is safe. L declined 25 of 30 (interval 0.664 to 0.927). This family should not be routed to a model at all. Whether a request may read the reserved test split is a permission, and permissions belong in code that checks the request before any model sees it. Routing is for work where a model's judgment is the job.

Reported metric. Both configurations are plausible, and the cost calculation decides: the small model if checking is cheap or automated, the large one if a person checks every row.

The family has to be known when the router decides. Whether a paper reports a metric often becomes clear only after the agent reads the table. Do the trusted lookup first, or enforce the stop after the tool returns, rather than asking a classifier to guess from the request's wording.

View full-size diagram

Run the routing study on your own agent

Keep a fresh holdout within each family for regression checks when a model, prompt or tool changes. Once you tune against a holdout case, it becomes development data; replace it before the next independent comparison.

Check the decision

What an interviewer would ask

Applied-AI teams choose between models constantly: a new snapshot, a cheaper tier, a distilled variant. An interview loop or a take-home that probes this skill might ask, "The cheaper model scores five points lower. Should we switch?" A strong answer asks what a rejected output costs, derives (c + g) / p, asks for the per-family breakdown, puts an interval on each acceptance rate, and names the family that should not be routed at all. A weak answer quotes a benchmark.

What would prove this wrong

The calculation assumes rejected attempts are caught and retried independently. If your grader misses errors, p is inflated and the cheap model looks better than it is; audit a sample of accepted rows by hand. If failures repeat on the same case, retries buy less than 1/p suggests; measure the retry acceptance rate directly. And if production traffic has a family your cases do not cover, the study says nothing about it until you add that family with its own rule.

The course that will build this study end to end with real models is Research Engineering and Evaluation Science, in preparation.