Did Model A Really Beat Model B? Eval Variance, Measured

A model scores 94% today and 89% tomorrow. Same problems, same prompt, same settings. Which score would you trust when choosing a model for your agent?

Let’s look at what changed: only the sampling. Before touching the run selector below, predict whether one more run will settle the comparison. Then switch runs. Notice how easily a single score could become a confident claim in a team’s chat.

Rod Rivera ran GPT-6 Luna on the same 100 AIME problems ten times. Scores ranged from 88% to 98%. We’ll use his recorded experiment to work out what one run can tell us, why repeated attempts aren’t independent evidence, and how to compare systems without throwing away the pairing.

Experiment provenance: Rod’s OpenAI measurements were collected on 2026-09-30; the 3,000 requests cost USD 3.06 at list price. Every number below traces to a public file. The run, statistics and Colab notebook that recomputes them are in profrodai-resources. You can recompute the saved results without paying for new model calls. Chapter 16, Evaluation as measurement, derives the statistics from scratch.

Every difference GPT-6 Luna, reworded − GPT-6 Luna you could have reported by running each system once on the same 100 tasks, from the ten runs of each that were made. The shaded band is the paired, task-level answer from all ten.

GPT-6 Luna, reworded vs GPT-6 LunaGPT-6 Luna vs GPT-5.6 Luna

View full-size figure

With one run of each, 24% of the 100 possible comparisons put GPT-6 Luna, reworded ahead and 71% put it behind. Using all ten runs, paired by task, the difference is −2.3 points, with a 95% interval from −4.3 to −0.3.

runs of each system averaged: 1

The experiment

• Problems: 100 problems sampled (seed 20261001) from AIME 1983 to 2024, numbers 6 to 15 of each exam, where the easy openers are left out. Every answer is an integer from 0 to 999, so grading is exact.
• Systems: Three, all at reasoning effort medium with an 8,000-token output cap:
  • GPT-6 Luna, asked: Solve this problem. The answer is an integer from 0 to 999. Reply with only the final integer.
  • GPT-6 Luna with the same request reworded: Work out the following competition problem and give just the resulting whole number (0 to 999).
  • GPT-5.6 Luna with the original wording.
• Runs: Every system answered every problem 10 times. That is 3,000 requests, USD 3.06 at list price.

The reworded prompt was meant as a control. It asks for the same thing in different words, so Rod expected it to score the same as the original. It did not, and a later section deals with that.

One run is one draw

The figure at the top pairs every run of one system with every run of the other: 10 × 10 = 100 comparisons someone could have reported after running each system once.

For GPT-6 Luna against GPT-5.6 Luna, 88 of those 100 comparisons put GPT-6 ahead, 7 put it behind and 5 are ties. Using all ten runs, the difference is 4.6 points, and its paired interval excludes zero. So a single run usually gets the direction right here. Twelve times in 100 it shows a tie or the wrong sign, and the size it reports ranges from −3 points to +14.

Running a paired test on a single run does not rescue it. McNemar's test looks only at the problems where the two systems disagree, which is the right thing to do with one run of each. On the ten runs taken one at a time, it found a significant difference (p < 0.05) in 3 of them. A real 4.6-point gap went undetected seven times in ten, because 100 problems once each is not much evidence.

Drag the slider in the figure. Averaging more runs of each system narrows the spread of reported differences, and it narrows toward the task-level answer.

Why 1,000 attempts are not 1,000 samples

The textbook standard error for a pass rate treats every attempt as an independent coin flip:

with $T = 100$ tasks and $n = 10$ runs. That would put GPT-6 Luna's pass rate at 91.8% ± 1.7 points.

The attempts are not independent. A problem the model finds hard stays hard on the next run. Out of 100 problems, 66 passed all ten runs and one passed none. The unit that was sampled is the task, so the honest standard error treats each task's pass rate $\hat p_i = c_i / n$ as one observation:

For GPT-6 Luna that gives 0.0169, nearly twice the naive figure. The ratio has a name. Write the intraclass correlation $\rho$ for the share of the variance in single attempts that comes from differences between tasks. Then the variance of the mean is inflated by the design effect

GPT-6 Luna's ICC is 0.309, so $D = 1 + 9 \times 0.309 = 3.78$ and $\sqrt{D} = 1.94$, the ratio between the two standard errors. The 1,000 attempts were worth about $1000 / 3.78 = 265$ independent samples.

100 tasks, each run n times. Attempts on one task are correlated (the ICC), so the run is worth fewer independent samples than it has attempts. Pick a measured system's ICC or drag your own.

GPT-6 Luna (ICC 0.31)GPT-6 Luna, reworded (ICC 0.28)GPT-5.6 Luna (ICC 0.20)

View full-size figure

100 tasks × 10 runs is 1,000 attempts. With an ICC of 0.31, the design effect is 1 + (10 − 1) × 0.31 = 3.78, so the run is worth about 265 independent samples, and an interval that treats every attempt as independent is 1.94× too narrow. As runs grow, the worth approaches 100 / ICC = 324: more tasks, not more runs, is what tightens it.

runs per task, n: 10intraclass correlation: 0.31

The effective sample is $nT / (1 + (n-1)\rho)$, and as $n$ grows it approaches $T/\rho$. For this suite that ceiling is $100 / 0.309 = 324$. A million runs per problem would still be worth fewer than 324 independent samples. If you want a tighter interval, add tasks. More runs of the same tasks mostly buy you a better estimate of each task's pass rate.

pass@k and pass^k, per task

Two numbers describe what repeated attempts give you. pass@k is the chance that at least one of $k$ attempts succeeds, which is what matters when a verifier can pick the good one. pass^k is the chance that all $k$ succeed, which is what matters when a user will run the same request again and expect the same result.

For a single task with success rate $p$ and independent attempts,

With $n$ recorded attempts, $c$ of them successes, you can estimate both without assuming a $p$. Draw $k$ of the $n$ attempts without replacement. The draw has no success with probability $\binom{n-c}{k}/\binom{n}{k}$ and has only successes with probability $\binom{c}{k}/\binom{n}{k}$, so

Both are unbiased for $k \le n$.

The mistake is to compute these from the suite's average pass rate. The average hides the tasks. Since $(1-p)^k$ and $p^k$ are convex in $p$, Jensen's inequality gives

Hard tasks drag pass@k below the average-based curve, and easy tasks hold pass^k above it. The measured gap is large. From GPT-6 Luna's 91.8%, the formula predicts pass^10 of $0.918^{10} = 42.5\%$. Measured task by task, it is 66.0%. At the other end, the formula predicts pass@10 of essentially 100%. It was 99.0%, because one problem failed all ten times, and pass@k never recovers a task the model cannot do.

Left: how many of its 10 runs each of the 100 tasks passed. Right: the chance that at least one of k attempts passes (pass@k) and that all k do (pass^k), measured, against the dashed curves that assume every task is equally hard.

GPT-6 LunaGPT-6 Luna, rewordedGPT-5.6 Luna

View full-size figure

View full-size figure

GPT-6 Luna: 66 of 100 tasks passed all 10 runs and 1 passed none. At k = 5, pass@k is 98.7% (the equal-difficulty curve predicts 100.0%) and pass^k is 76.1% (it predicts 65.2%). The gap from the dashed curves is the spread in difficulty between tasks.

attempts, k: 5

The histogram on the left of the figure is why. Two thirds of the tasks sit at 10 of 10, most of the rest at 8 or 9, and one at 0. A suite like that has a high pass^k and a pass@k that stops short of 100%. Estimate both per task, average them, and say which one your use case needs.

Paired differences and a bootstrap over tasks

To compare two systems, run both on the same tasks and difference them task by task:

Pairing removes the difficulty that the two systems share. A hard problem is hard for both, so it adds to both pass rates' variance and mostly cancels in $d_i$. For GPT-6 Luna against GPT-5.6 Luna, treating the two as independent samples would give a standard error of $\sqrt{0.0169^2 + 0.0177^2} = 0.0245$. Paired, it is 0.0155.

For the interval, resample what was sampled. Draw 100 tasks with replacement, keep all ten runs of each system on each drawn task, recompute $\bar d$, and repeat 4,000 times. The 2.5th and 97.5th percentiles are the interval. Resampling individual attempts instead would treat them as independent again, and the interval would come out too narrow by the same factor $\sqrt D$.

The model upgrade is a real improvement on this suite, somewhere between 1.5 and 7.7 points. That is the claim you can make. "GPT-6 Luna is 14 points better," which the luckiest pair of single runs would have shown, is not.

The rewording that should have done nothing

Rod added the reworded prompt as a null control: same model, same request, different words. It came out 2.3 points worse, with an interval of −4.3 to −0.3.

Check the formatting explanation before accepting it. Replies the API marked incomplete were 12 of 1,000 for the reworded prompt and 9 for the original, and every answer was parsed the same way. The loss is spread thin: the reworded prompt did worse on 33 problems, better on 19, and the same on 48, mostly by one run in ten either way.

Two readings fit. The wording may matter. "Competition problem" and "whole number" are not the same request as "solve this problem" and "integer", and a model can respond to that. Or it is chance: this experiment made two comparisons, this one's p-value is about 0.02, and a correction for two comparisons leaves it right at the edge. We should rerun the pair on new problems before believing either reading. In practice the lesson is the same both ways. A wording change you think is cosmetic can move a score by as much as the run-to-run noise, so treat prompt wording as a variable. Measure it, and fix it before you compare anything else.

What would change these conclusions

• Math with exact answers: AIME grading has no noise, and the problems are short. Agent tasks with long trajectories, flaky tools and judged outputs add variance sources that this run does not have (a judge's own error, for one, which Calibrate an LLM Judge Before You Trust Its Score measures), so expect larger design effects there, not smaller.
• Public problems: AIME 1983 to 2024 is almost certainly in both models' training data. That can make some problems easy for reasons unrelated to reasoning, which would inflate the share of tasks at 10 of 10.
• One effort level, one cap: Everything ran at medium reasoning effort with an 8,000-token cap. At other settings the ICC, the pass curves and even the ranking could differ.
• One run of the whole experiment: The tasks were sampled once. The bootstrap covers that sampling; it does not cover a model update after 2026-09-30.

Why this matters in a lab

Frontier labs decide on checkpoints, prompts and training changes from evaluation deltas of a few points, measured on suites where the same tasks are run many times. Every one of those decisions needs a standard error that respects the tasks, pass metrics computed per task, and a paired comparison with an honest interval. People who can produce those, and who can say why a 3-point gain on one run means little, are the ones whose numbers get trusted.

An interviewer might ask: your new checkpoint scores 2 points higher than the old one on a 200-task suite, each task run 5 times. Is that an improvement?

Investigate one comparison

Now make a small research artifact of your own. Use the linked notebook’s saved results; leave the model-calling cells alone unless you deliberately choose to pay for a new experiment.

1. Write your prediction before recomputing: will the model difference survive a paired interval? Which rows should a bootstrap resample?
2. Recompute the paired difference and interval from the recorded rows. Keep all runs of a sampled task together. Record your output and the dataset/source revision.
3. Write a one-page memo: claim, sampling unit, interval, negative control and one limit. Explain why another run of the same tasks is different from adding new tasks.
4. Read one opportunity below. Find the employer’s measurement responsibility, then identify a requirement your memo does not demonstrate. Choose a next exercise for that gap.

This memo is evidence of a measurement exercise. It is not a hiring credential or proof that you meet an employer’s requirements. A paper or a NeurIPS publication is not a universal prerequisite for this work; read the actual listing.