Read a Coding Benchmark Score: Contamination, Validity and Error Bars

The score that answered a different question

A new checkpoint resolves 72% of a 500-task coding benchmark. The model your team uses today resolves 70%. Your team runs a coding agent inside its own training-code repository, and someone asks whether to switch. The benchmark number is real evidence. It is evidence about a specific sample of tasks, graded by a specific oracle, on data the model may have seen. Before it moves a decision, you need to know how large its error bar is, whether the tasks are clean, and whether the oracle is right.

This article works through each question with the arithmetic shown. The numbers in the worked examples are hypothetical unless a source is cited. Chapter 16 derives the interval and the paired test used here. Chapter 17 derives the winner's curse, which explains why the top of any leaderboard is optimistic.

What a coding benchmark measures

In a benchmark built from GitHub issues, each task gives the model a repository and an issue, the model proposes a patch, and the repository's tests decide pass or fail. SWE-bench built 2,294 such problems from real repositories and graded them by whether the patch passes the tests. That oracle is cheap, deterministic and easy to report. It also checks only what the tests check.

Three things stand between that score and your decision:

1. Sampling error. The score is an estimate from a finite set of tasks.
2. Validity. The tasks may be contaminated, and the oracle may accept wrong patches or reject right ones.
3. Transfer. Your repository, tools and failure costs differ from the benchmark's.

Error bars on a leaderboard number

Treat each task as a pass or fail with unknown rate p. With n tasks the standard error of the score is √(p(1 − p)/n). For p = 0.70 and n = 500:

SE = √(0.70 × 0.30 / 500) = √0.00042 = 0.0205.

A 95% interval is about ±1.96 × 0.0205 = ±0.040, so 66% to 74%. A two-point gap between two models sits well inside that.

The two models ran on the same 500 tasks, so compare them paired. Only tasks where they disagree carry evidence. Suppose 50 tasks are discordant: the new checkpoint alone passes 30, the old model alone passes 20. Under "equally good," each discordant task is a fair coin, and McNemar's exact test gives p = 0.20. A 30 to 20 split happens by chance one time in five. The two points are not established.

Why the top of the board is optimistic

A leaderboard ranks many models and you look at the best. Chapter 17 shows that if k candidates are equally good and each score has noise σ, the winner's measured score exceeds its true score by σ times the expected maximum of k standard normals, which is 1.766 for k = 16. With σ = 0.0205, the best of 16 equally good models would look 1.766 × 2.05 ≈ 3.6 points better than it is. Rerun the chosen model on tasks it never competed on before you believe its rank.

Contamination: has the model seen the tasks?

Tasks built from public repositories may be in a model's training data, together with their fixes. Scores on seen tasks measure recall as much as skill. Two tests help.

A time split. Divide tasks by the date the issue was created, before and after the model's training cutoff. Hypothetically, the model resolves 180 of 250 older tasks (0.72) and 150 of 250 newer ones (0.60). The difference is 0.12, with standard error

√(0.72 × 0.28 / 250 + 0.60 × 0.40 / 250) = √(0.000806 + 0.000960) = 0.042,

so a 95% interval of 0.12 ± 0.082, or about 4 to 20 points. The gap is real at this sample size. Its cause is not identified: newer tasks may also be harder, or come from different repositories. Match the two halves on repository and difficulty before calling the gap contamination.

A fresh replication. Write new tasks that follow the benchmark's recipe and have never been public, then compare scores. Zhang et al. (2024) did this for grade-school math with GSM1k and reported evidence of systematic overfitting in several model families, while frontier models showed little. The design, not their numbers, is what transfers: a fresh, comparable set is the cleanest contamination test available.

Oracle validity: do the tests grade correctly?

Tests can accept a wrong patch when they are too weak, and reject a correct one when they are too specific to one implementation. Measure it. Have two engineers review a random sample of passing patches, blind to the model. Suppose they judge 34 of 40 correct. The oracle's precision on passes is 0.85, with a Wilson 95% interval of about 0.71 to 0.93. A corrected resolved rate is then roughly 0.70 × 0.85 = 0.595, and its interval must carry both sources of error. Review failing patches too, since an overly strict test lowers the score for the opposite reason.

The code below reproduces every number in the last three sections with the standard library. It runs on Python 3.12, including Colab.

from math import comb, sqrt


def mcnemar(b, c):
    m = b + c
    return min(1.0, 2 * sum(comb(m, i) for i in range(min(b, c) + 1)) / 2**m)


def wilson(k, n, z=1.959963984540054):
    p = k / n
    d = 1 + z * z / n
    center = (p + z * z / (2 * n)) / d
    half = z / d * sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return center - half, center + half


se = sqrt(0.70 * 0.30 / 500)
print(f"score 0.70 on 500 tasks: SE {se:.4f}, 95% half-width {1.96 * se:.3f}")
print(f"McNemar, 30 vs 20 discordant tasks: p = {mcnemar(30, 20):.3f}")
print(f"winner's curse, best of 16 equal models: +{1.766 * se:.3f}")
gap_se = sqrt(0.72 * 0.28 / 250 + 0.60 * 0.40 / 250)
print(f"time split 0.72 vs 0.60: SE {gap_se:.4f}, interval [{0.12 - 1.96 * gap_se:.3f}, {0.12 + 1.96 * gap_se:.3f}]")
low, high = wilson(34, 40)
print(f"oracle precision 34/40: [{low:.3f}, {high:.3f}], corrected rate {0.70 * 34 / 40:.3f}")

score 0.70 on 500 tasks: SE 0.0205, 95% half-width 0.040
McNemar, 30 vs 20 discordant tasks: p = 0.203
winner's curse, best of 16 equal models: +0.036
time split 0.72 vs 0.60: SE 0.0420, interval [0.038, 0.202]
oracle precision 34/40: [0.709, 0.929], corrected rate 0.595

Build a local mapping before the number moves a decision

A benchmark still has a role. It can shortlist models worth testing when coding ability matters to your task. It cannot fill in your own oracle or your own critical failure. Write down, for the benchmark and for your task, the same five fields:

The last row is the one no leaderboard supplies. A patch that quietly changes how an evaluation metric is computed can pass every test and corrupt every result that follows.

This site has not run this comparison on real models. When a model decision rests on it, the protocol above is what to run and what to publish.

Agreement, recall and coverage are three different numbers

When free-form output such as a code review is graded by a model judge, a second collapse happens. Agreement is the share of transcripts where judge and human labels match. Per-family recall is the share of labeled failures in one family that the judge catches. Coverage is which failure families have any test cases at all. "Our judge is 90% reliable" usually reports the first while readers hear the third. Agreement can be high on an imbalanced set while a critical family's recall is low, and a family with no cases has unmeasured recall, not zero. Kappa corrects agreement for chance but is still one blended number. The judge calibration article works all three with intervals.

View full-size diagram

View full-size diagram

What would prove this reading wrong

If a model's rank on a public benchmark reliably predicted its paired result on your own task set, across several model generations, the local evaluation would add little and you could lean on the leaderboard. That is a checkable claim: keep your local results next to the public ones and compute how often the order agrees. Until you have that record, treat the benchmark as a shortlist.

Check the decision

What an interviewer would ask

"Model A beats model B by two points on a coding benchmark. What do you conclude?" is a common question in applied-AI and research interviews, because the answer shows whether you treat a score as a measurement. A strong answer computes the standard error, asks whether the comparison was paired and counts the discordant tasks, names the winner's curse for a leaderboard, and proposes a contamination check and an oracle audit. Then it says what local evaluation would settle the actual decision. The upcoming research engineering and evaluation science course plans a module on designing evaluation sets and checking them for contamination before any system sees them.