Agent Evaluation Variance: Did A Really Beat B?

The one run that decided nothing

Picture two versions of a triage prompt for an agent that routes support tickets. Call them prompt A and prompt B. Someone runs both against a tricky enterprise-outage ticket. A produces a clean, correctly slotted response. B produces something a little slower and a little less precise. The team declares A the winner and ships it.

That single comparison tells you almost nothing about which prompt is actually better. Agents are probabilistic systems: the same prompt, same input, same model can produce a different trajectory on the next call. A "win" in one trial could reflect a genuine and repeatable difference between A and B, or it could reflect the ordinary variation you'd see if you ran the identical prompt against itself twice. Without repeating the comparison, you cannot tell those two situations apart, and the decision to ship A is really a decision made on a sample size of one.

This is the trap: a single selected run standing in for a claim about reliable performance. The fix is not complicated, but it does require discipline before you run anything — you declare your task families, your trial count, and your acceptance rule in advance, then you report what actually happened, including the failures.

What a repeated trial actually measures

Before working the example, it helps to be precise about three things that are easy to blur together: a task family, a trial, and independence.

A task family is a group of inputs that share a failure mode you care about — for instance, "enterprise-tier tickets reporting a production outage" is one family, and "billing questions with no urgency" is a different family, because a system can be reliable on one and unreliable on the other. Anthropic's engineering guidance on evaluating agents makes a related point: agent evaluation has to separate tasks, environments, trajectories and outcomes, because a single aggregate score cannot stand in for every operating requirement an agent needs to meet (Anthropic, "Demystifying evals for AI agents", retrieved 2026-09-10). A prompt that scores 90% overall but 0% on the outage family is not a 90% prompt for a support desk; it is a prompt that fails exactly where failure is most expensive.

A trial is one execution of a prompt against one input from a family. Ten trials of the same input are not ten independent tests of the prompt's general competence — they are ten repeated attempts at the same specific case, useful for measuring how stable the model's behavior is on that case, but not a substitute for testing across different cases.

Independence is an assumption about how observations relate; it needs justification for the intended statistical analysis. Ten trials on the same outage ticket tell you about variance on that one ticket. Ten trials spread across ten different outage tickets tell you something closer to variance across the family. Conflating the two is one of the easiest ways to overstate confidence in a result, because "thirty runs" sounds like thirty pieces of evidence even when it's really repeated calls clustered around a small number of cases.

Designing the paired comparison

Here is a bounded, worked design for comparing prompt A against prompt B, built the way you'd actually plan it before running anything.

Freeze three task families first: an enterprise-blocking-outage family, a routine billing-question family, and a critical-policy family (say, "must never recommend a competitor's product by name," echoing the kind of drafting failure described in an existing course case on evaluation engineering across support-agent layers — see the worked lesson on building eval suites across both architecture quadrants). For this small illustration, choose five distinct case IDs per family and two repetitions per case, giving ten trials per family per prompt. Pair A and B by case ID, repetition, environment snapshot and grading rule; randomize or alternate their execution order. That gives you a 3-by-10 outcome table for A and another for B, sixty labeled cells total, not sixty independent measurements of "which prompt is better."

Here is what a reporting table looks like, filled in with clearly labeled hypothetical results (not an executed measurement):

The totals would be A: 26/30, about 86.7%, and B: 18/30, 60%. Those summaries are valid arithmetic but insufficient for promotion. That overall number buries a family where B fails completely. A zero-of-ten result in a critical-policy family is disqualifying on its own, regardless of what happens elsewhere, and averaging it away is exactly the failure this design is meant to prevent.

The marginal counts do not show which paired trials differed. In the outage family, both of these paired tables produce A = 9/10 and B = 8/10:

Retain case-level pairs, not just totals, so analysis can use the disagreements. Repeated calls to one case may be independent conditional on a fixed environment, but they still estimate performance on that case. Shared sessions, caches or infrastructure may introduce dependence. For uncertainty across the intended workload, sample diverse cases and use an analysis that respects repetitions clustered within cases.

Separating model variation from infrastructure noise

There's a second confound worth naming, and it often appears in operational testing: not every difference in outcome comes from the model. Timeouts, queue contention, rate limits and cold starts can make identical model output look like a failure, or make a slow success look like an outright loss in a race-conditioned harness.

The way to isolate this is to run a control pass first: replay identical, fixed, pre-recorded fixture responses through both harnesses (A's and B's), with no live model call at all, and measure how often the harness itself times out, queues, or drops a request. Only after that control pass do you run the live paired trials. If the fixture-replay control shows, say, 2 of 20 harness-level failures with no model involved at all, you have evidence of infrastructure failures in that control. It does not identify the cause of any particular live failure, and you report those separately rather than folding them into "model variance."

This distinction matters because an unbalanced execution environment can confound the comparison. If B was tested on a different day under heavier load, or against a different task mix than A saw, you no longer have a paired comparison — you have two unrelated measurements that happen to sit in the same table.

Steps for running the comparison honestly

What would falsify the result

A result like "A wins" only holds up if you can say what would have shown otherwise. Here, that means: if repeating the ten trials per family with a different random seed, or on a different day, produces materially different family-level pass counts, the original estimate needs a wider uncertainty assessment. Sampling variation can coexist with a real difference, so changed counts alone do not prove the original effect was entirely noise. It also means that if the fixture-replay control shows a meaningfully different harness failure rate between A's and B's execution environments, any apparent model difference is confounded and the comparison needs to be rerun with matched infrastructure before it means anything.

A separate warning applies to the benchmarks you might be tempted to borrow instead of building your own. A recent preprint on SWE-Bench Pro Verified examines reliability problems in coding-agent benchmarks, including task leakage and quality issues in the underlying tasks (SWE-Bench Pro Verified preprint, retrieved 2026-09-10). That paper is a research lead about one benchmark family, not a general verdict on coding agents, and it is a useful reminder that a public benchmark's aggregate score carries the same risk as your own averaged table: a single number can mask exactly the family-level failure that matters for your task.

Turning the table into an acceptance decision

For a team actually deciding whether to ship, the acceptance rule has to be written before the numbers come in, not fitted afterward. One reasonable rule: every critical-policy fixture must pass its deterministic check (in the competitor-mention example, an exact string check is sufficient only for the declared literal-name fixture; aliases, spelling variants and contextual recommendations need broader cases), and any family with a zero-pass result blocks promotion regardless of the aggregate. Unresolved or ambiguous outcomes route to human review rather than getting silently counted as a pass.

Under that rule, neither prompt is promoted: A fails three critical trials and B fails ten. A has the better observed totals, but it still misses the predeclared all-critical-cases requirement. The acceptance record should name the specific failing case IDs and the owner responsible for fixing them — not just record a single blocking verdict and move on.

Check the decision

What to build next

The worked table above uses clearly labeled hypothetical numbers to teach the design. Before this method means anything for your own system, you have to build the fixture set, freeze the trial count, and actually run it — and only report family-level results you can point to, not ones assumed from the shape of the exercise. If your agent already spans multiple layers of a pipeline, from a conversational front end down to a research or extraction layer, the same family-level discipline applies at each layer separately, since a layer that saturates an easy family can still fail badly on a rare, high-stakes one.