LLM Model Routing Evaluation From Accepted Work Cases

LLM model routing evaluation begins with your tasks

Suppose you are deciding whether to route a shop-replenishment assistant's requests to a smaller, cheaper model or keep everything on a larger one. You look up a benchmark leaderboard and see the larger model scores higher on average. That number answers a question you did not actually ask. Your assistant does not process "average tasks." It processes ordinary stock checks, requests where a price is missing from the catalog, and requests where a shop's policy conflicts with what a customer wants. Those three situations fail in different ways, at different rates, under different configurations. An aggregate score blends them into a single figure that cannot tell you which configuration to trust for which situation.

Model routing means choosing among configurations: given several possible configurations (which model, which sampling settings, whether a planner and an executor are separated), pick a rule that assigns each incoming task to one of them. In this example, the routing rule maps a task family to a configuration. Other routers may use task features, measured difficulty or confidence; whichever signals they use must be available at decision time. Anthropic's engineering writeup on evaluating agents makes a related point about agent evaluation generally: it separates tasks, environments, trajectories, and outcomes, and argues that a single aggregate cannot establish every operating requirement (Anthropic, "Demystifying evals for AI agents"). Routing decisions inherit that same structure: you need per-family evidence, not one number.

Freezing the task families before comparing anything

Define the operational cases first. Before running any comparison, write down the families of task your system will actually see, and write an acceptance rule for each one independently. For the replenishment assistant, three families do most of the work:

Notice that "success" means something different in each row. A correct draft order is success for the first family. A refusal with a clear explanation is success for the second and third. If you measure only "did the assistant produce a usable order," you will score the missing-price and policy-conflict families as failures even when they behaved exactly as required, or worse, you will reward a configuration that fabricates a price to keep the completion rate high. This is the same discipline the planner-executor teaching case describes: a reviewer needs source facts and a specific acceptance test, not a general sense that the output "looks done" (see the model-choice and planner-executor lesson).

Running the comparison without collapsing it into one score

With the families fixed, compare configurations on the same inputs, tool access and acceptance rules. Set the release thresholds before inspecting results. In this hypothetical pilot, drafts remain subject to human review: ordinary-stock cases require at least 11 correct drafts out of 12, while missing-price and policy-conflict cases each require 6 correct stops out of 6. These are illustrative acceptance thresholds for this small fixture, not estimated production reliability.

The table contains invented results to work through the decision. It compares whole configurations: changing both model size and planner structure does not isolate the effect of either change.

If you only read the mean cost column, Route C looks attractive. But Route C fails the missing-price family badly: in four of six hypothetical cases it would have produced a number instead of stopping. That fails the declared missing-price rule. Route A has the strongest ordinary-stock and missing-price behavior but the worst policy-conflict handling, missing half of those cases in this illustration. Route B alone meets all three declared fixture thresholds: 11/12, 6/6 and 6/6. Its one ordinary-stock error still needs investigation and human correction under this pilot's operating rule. Passing this small fixture is permission to continue evaluation, not proof that the configuration is safe for unattended purchases.

This is the acceptance rule in practice: a candidate route cannot pass solely on aggregate accuracy when one critical family regresses. You reject Route C outright, not because its average is worse (it might not be) but because a named family failed against its own rule. You would also reject a hypothetical Route D that matched Route B everywhere except for a single missed policy-conflict case, since "one worse" is not "acceptable" when the rule is a hard stop.

The diagram assumes the family has already been established. In an actual workflow, a missing price may become apparent only after a catalog lookup. Perform that trusted lookup before routing on price availability, or enforce a stop after the tool returns. Do not assume a classifier can infer unseen catalog state from the request wording.

Evaluate the router together with the selected configuration. A model that handles missing prices correctly provides no protection if the router sends those cases elsewhere. Test ambiguous requests and combinations of conditions, such as a missing price on an item that also violates policy. If the family cannot be established from trusted inputs, use a documented review path rather than silently selecting the cheapest route.

Why the fallback question is not a cost question

Suppose someone proposes Route C because it costs less. The missing-price rule still requires stopping instead of inventing a number. That requirement must hold in the complete routed system, including fallbacks and tool failures.

Mixed routing can be useful: select the cheapest configuration that meets the declared criteria for a particular family. In this table Route C also misses the ordinary-stock threshold, so the evidence does not yet justify assigning it that family either. A revised router is a new candidate. Test its classification, selected configurations and fallback path together before applying the earlier results to it.

Building the comparison as a repeatable procedure

The comparison above only means something if it is run the same way every time a route changes. Treat it as a fixed procedure rather than an ad hoc spreadsheet.

That third step deserves a concrete artifact. A rejected-route record for the hypothetical Route C above would read something like: "Route C rejected. Failed family: missing price. Failing cases: 4 of 6 in the illustrative comparison. Acceptance rule unchanged: system must stop and report, never estimate. Operating fallback: route missing-price tasks to Route B pending further testing." Writing the rejection down this way keeps a future reviewer from re-proposing the same route because "the average looked fine" a few months later.

Guarding a route against silent regressions

A route can regress when a model, prompt or tool changes. Reserve fresh cases within each known family and keep their expected answers outside the tuning process. Those cases test whether the candidate still handles the same operating requirements on inputs it was not optimized against. Predeclare the run count and conditions; nondeterministic agents can pass once and fail on a repeat.

An entirely held-out family asks a different question: can the system transfer to a new kind of task? For example, keep bulk-discount requests separate while developing ordinary-stock and missing-price behavior. Freeze that family's acceptance criteria before exposing it, then evaluate the complete routed path. This complements within-family regression tests; it does not replace them or provide the only way to detect regressions.

Once developers inspect a failed holdout and tune against it, that case becomes useful development or regression material. It is no longer untouched evidence. Preserve it, record the change and obtain fresh holdout cases for the next independent comparison. Repeatedly tuning until one fixed holdout passes would recreate the overfitting the split was intended to prevent.

Keep configuration versions, individual case outcomes and actual cost components in the same record. The table's "Higher" and "Lower" labels are placeholders for the illustration. A real comparison should count failed attempts, retries, tool calls and human review under the same costing convention, then divide by accepted outcomes when that is the decision being priced.

Check the decision

What a route comparison cannot tell you

A comparison built this way is bounded by what it tests. If your three families do not cover a fourth situation your assistant will actually meet, such as a customer requesting a bulk discount that touches both pricing and policy at once, the comparison says nothing about that case until you add it as its own family with its own acceptance rule. A comparison run on twelve, six and six hypothetical cases per family, as sketched above, is also small; real deployment volumes will surface edge cases a table this size cannot contain. Treat every comparison table as current evidence for the families it actually tested, and extend it the moment your system meets a task type it was not built to grade.

The worked table, the rejected-route record and the held-out family are proposed practices to build and check, not a report of an executed production run. An accepted or rejected route should trace back to a configuration version, a case, a family and a rule that another engineer can inspect and rerun.