Test an Agent's Fallback Path Against the Same Acceptance Bar

The fallback that parses and still fails

An agent reads papers and technical reports and fills rows in a team's comparison table. Its primary model is rate limited during a busy afternoon, so the harness falls back to a smaller model. The fallback returns this row:

{"benchmark": "long-context-qa", "metric": "accuracy", "value": 0.713, "status": "ok", "source_ref": null}

It is valid JSON with the right keys. A dashboard that checks "did we get a well-formed row" shows green. But the paper never reports that benchmark. The number 0.713 came from nowhere. If it lands in the comparison table, it looks exactly like a measured result, and someone may cite it in a design review.

The primary configuration is required to answer needs_data when the paper does not contain the number. The fallback is held to the same rule. Switching models does not change what counts as an acceptable row. A degraded mode can promise less, such as "draft only, never written to the table", but that promise has to be declared and tested. It does not follow from the switch.

This article is about the evaluation problem underneath: a fallback runs rarely, so the usual measurements barely see it.

Why production traffic cannot measure a fallback

Let a be the fraction of requests the primary handles, p1 the primary's acceptance rate and p2 the fallback's. Every request goes to exactly one path, so the overall acceptance rate is a weighted average:

P(accepted) = a · p1 + (1 − a) · p2

Put in numbers. These are illustrative, chosen to show the arithmetic: the primary handles 98% of requests and is accepted 94% of the time; the fallback is accepted 70% of the time.

P(accepted) = 0.98 × 0.94 + 0.02 × 0.70 = 0.9212 + 0.0140 = 0.9352

The fallback drops overall acceptance from 0.94 to 0.935. Half a point. If the fallback accepted nothing at all, overall acceptance would be 0.921. A dashboard that tracks the aggregate cannot tell a decent fallback from a useless one.

The sample is the second problem. At 2% engagement, 10,000 requests give the fallback about 200 trials. Those trials are not a random sample of your work. They arrive clustered in outages and rate-limit spikes, when traffic is unusual and other dependencies may be degraded too. A fallback measured only in production is measured late, on a biased sample, and diluted in the average.

Force failover on the eval set

Measure the fallback directly. Run your eval set with the primary disabled, so every case goes through the fallback path: same inputs, same tools, same grader.

How many cases? To estimate an acceptance rate p to within ±5 points at 95% confidence, solve 1.96·√(p(1−p)/n) = 0.05 for n:

n = 1.96² · p(1 − p) / 0.05²
p = 0.5:  n = 3.8416 × 0.25 / 0.0025 ≈ 384
p = 0.8:  n = 3.8416 × 0.16 / 0.0025 ≈ 246

So a few hundred cases pin the fallback down. That is a normal eval run, and it takes an afternoon rather than months of waiting for outages. If a forced-failover run accepted 140 of 200 cases, the 95% Wilson interval would be 0.633 to 0.759. Chapter 16 derives that interval and explains why it behaves better than the textbook one near 0 and 1.

Grade each case against the rule for its family, decided before you run anything:

Report each family separately. The fixture above fails the missing-metric family even though it parses, because the required status is needs_data and the actual status is ok. A grader that only checks parseability passes it.

View full-size diagram

Before trusting the grader, give it controls. A needs_data row with no value must be accepted in the missing-metric family; change only its status to ok and the verdict must flip. This checks the grader's decision boundary without calling any model.

View full-size diagram

Compare the fallback to the primary, pair by pair

You now have two graded runs over the same cases: primary and forced fallback. Compare them case by case, as in the routing study. The cases both pass or both fail say nothing about the difference. The discordant cases do: those only the primary passes show what failover costs you, and they tell you which family to restrict.

Three outcomes are common:

• It meets the bar on every family: ship it as a true fallback.
• It meets the bar on reported metrics but fabricates on missing ones: restrict the degraded mode. Let it draft reported-metric rows for review, and send missing-metric cases to a queue instead of the table.
• It fails the held-out-split rule: that rule should not depend on a model in the first place. Move the permission check into code that runs before either model sees the request.

The book's Chapter 14 measured another fallback-specific failure: a model that answers in words instead of calling a tool. In its runs, every model did it sometimes, from 5 to 33 times in 216 native answers. In an agent loop, a text answer ends the turn, so a weaker fallback can quietly skip the lookup that would have told it the metric is missing. Count answers with no tool call as their own row in the fallback's results.

A forced-failover protocol you can run

Anthropic's engineering writeup on agent evaluation makes the general case for this structure: separate tasks, trajectories and outcomes, and grade with domain-specific criteria rather than one aggregate score (Anthropic, "Demystifying evals for AI agents," https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents). A fallback changes the configuration answering the task. The task's criteria stay the same.

Check the decision

Why this matters in a lab

Research and applied teams run agents against rate-limited APIs, shared inference clusters and preview models that come and go. Fallbacks are everywhere, and they fail when the system is already stressed. A team that owns an agent in production will expect you to know that a rarely used path is barely measured by traffic, that its failures hide in the aggregate, and that the fix is a designed experiment: force the condition, size the sample, grade against the same contract. An interviewer might ask, "Your fallback model has never caused an incident. Is it safe?" The answer starts with the weighted average and ends with a forced-failover run.

What would prove this wrong

If your fallback engages often, say a third of the time, production traffic measures it well and the aggregate reflects it; the argument here is about rare paths. If your eval set does not resemble outage-time traffic, a forced-failover run on it can pass while real failover fails; add cases drawn from past incidents. And if the fallback is nondeterministic, one pass per case is one trial. Repeat the missing-metric family, knowing from Chapter 16 that repeats measure variability on a case, not new cases.