The pass rate that went up because the model timed out
You run a 400-case eval. Twenty cases time out and return no answer. Of the 380 that answer, 300 are correct. What is the pass rate?
If the grader divides by every case in the split, it is 300/400 = 75.0%. If it divides by answered cases, it is 300/380 = 78.9%. The second number rewards the model for failing to answer. The 95% half-width at p = 0.75 and n = 400 is 1.96·√(0.75 × 0.25 / 400) = 1.96 × 0.0217 = ±4.2 points, so a one-word change in the grader moved the headline by almost the whole error bar. Nothing crashed. Every test that only checked "the grader returns a number between 0 and 1" still passed.
A grader is the code that produces your result. It deserves the same treatment as any code whose output you publish: a written policy, boundary cases whose expected answers come from that policy, and proof that each case can actually fail. This article builds all three for a small numeric grader and then breaks the grader on purpose seven ways.
From one policy paragraph to a table
Start with the policy, in words a colleague can check:
A case passes when the answer's final number is within 1% of the authored answer, measured relative to the authored answer's magnitude. A missing or non-numeric answer fails. The pass rate divides by every case in the split. A case answered twice is an error. An answer for a case not in the split is rejected.
Each sentence is a separate way the grader could be wrong, so each gets at least one row. The expected results below were written by hand from the policy, before running anything.
| Row | Setup | Expected result | What it pins down |
|---|---|---|---|
| Tolerance edge | Truth 400, answer 404 | pass | The comparison is <=, not < |
| Just over | Truth 400, answer 404.5 | fail | The tolerance is 1%, not looser |
| Negative truth | Truth −50, answer −50.4 | pass | Tolerance uses the magnitude of the truth |
| Missing answer | Four cases, one unanswered, one wrong | 2 passed of 4 | Missing fails; denominator is all cases |
| Duplicate case | Same case answered twice | duplicate_case | Repeats are not silently merged |
| Unknown case | Answer for a case not in the split | unknown_case | Answers cannot add cases |
| Boolean answer | Truth 1, answer True | fail | True == 1 in Python must not pass (why) |
| String answer | Truth 400, answer "400" | fail | Extraction failures are failures |
Scroll horizontally to view columns.
The arithmetic for the edge rows: 1% of 400 is 4, and |404 − 400| = 4 ≤ 4, so it passes; |404.5 − 400| = 4.5 > 4, so it fails. For the negative row, 1% of |−50| is 0.5 and |−50.4 − (−50)| = 0.4 ≤ 0.5. A grader that wrote rel_tol * truth without the absolute value would compute a tolerance of −0.5, and no answer could ever pass a negative truth.
Why the oracle has to be independent
Here is the trap the table is built to avoid. Suppose the grader divides by answered cases, and the test suite computes its expected pass rate with a helper that also divides by answered cases. The grader and the helper agree on 78.9%. The test passes. The bug ships, because the oracle was never independent of the thing it checked. It was a mirror.
Derive expected values from the policy text and known inputs, without importing or calling the implementation. A second person can help catch shared assumptions, but a second person is not proof of independence on its own; one careful engineer can derive values independently, and two people can copy the same wrong formula.
The LangChain team's post on building agent environments and tasks (Vivek Trivedy and Nick Hollon) makes a related structural point: a task needs an input, an environment and a separate way to judge the result, kept apart so judgment does not fold into the thing being judged. The hand-computed column is that separate judge for your grader.
Prove every row can fail: mutation testing
A table you cannot break is a table you cannot trust. Mutation testing checks the tests themselves: make one small, deliberate change to the grader (a mutant), run the table, and confirm that at least one row fails. A mutant no row catches has survived, which means some part of the policy is untested.
Here is the grader, the table and seven mutants. It runs on Python 3.12 with the standard library.
import inspect
def grade(answers, expected, rel_tol=0.01):
ids = [case_id for case_id, _ in answers]
if len(ids) != len(set(ids)):
return "duplicate_case"
if not set(ids) <= set(expected):
return "unknown_case"
given = dict(answers)
passed = 0
for case_id, truth in expected.items():
answer = given.get(case_id)
if isinstance(answer, (int, float)) and not isinstance(answer, bool):
if abs(answer - truth) <= rel_tol * abs(truth):
passed += 1
total = len(expected)
return {"passed": passed, "total": total}
# Expected results are written by hand from the policy, not by running grade().
FOUR = {"a": 10, "b": 20, "c": 30, "d": 40}
TABLE = [
("tolerance edge", {"a": 400}, [("a", 404)], {"passed": 1, "total": 1}),
("just over", {"a": 400}, [("a", 404.5)], {"passed": 0, "total": 1}),
("negative truth", {"a": -50}, [("a", -50.4)], {"passed": 1, "total": 1}),
("missing answer", FOUR, [("a", 10), ("b", 20), ("c", 31)], {"passed": 2, "total": 4}),
("duplicate case", {"a": 400}, [("a", 400), ("a", 400)], "duplicate_case"),
("unknown case", {"a": 400}, [("z", 400)], "unknown_case"),
("boolean answer", {"a": 1}, [("a", True)], {"passed": 0, "total": 1}),
("string answer", {"a": 400}, [("a", "400")], {"passed": 0, "total": 1}),
]
MUTANTS = {
"strict tolerance": ("<= rel_tol", "< rel_tol"),
"signed tolerance": ("rel_tol * abs(truth)", "rel_tol * truth"),
"absolute tolerance": ("rel_tol * abs(truth)", "rel_tol"),
"answered denominator": ("total = len(expected)", "total = len(given)"),
"missing counts as pass": ("given.get(case_id)", "given.get(case_id, truth)"),
"no duplicate guard": ("if len(ids) != len(set(ids)):", "if False:"),
"Boolean is a number": (" and not isinstance(answer, bool)", ""),
}
def failing_rows(grader):
return [name for name, expected, answers, want in TABLE
if grader(answers, expected) != want]
assert failing_rows(grade) == [], "the correct grader must pass every row"
source = inspect.getsource(grade)
for mutant, (old, new) in MUTANTS.items():
assert old in source, mutant
namespace = {}
exec(source.replace(old, new, 1), namespace)
caught = failing_rows(namespace["grade"])
print(f"{mutant:22} caught by: {', '.join(caught) or 'NOTHING (survived)'}")Run on 2026-09-30 with Python 3.12.13, it printed:
strict tolerance caught by: tolerance edge
signed tolerance caught by: negative truth
absolute tolerance caught by: tolerance edge, negative truth
answered denominator caught by: missing answer
missing counts as pass caught by: missing answer
no duplicate guard caught by: duplicate case
Boolean is a number caught by: boolean answerSeven mutants, seven caught, and each by the row written for it. Now delete the tolerance-edge row and run again: the strict-tolerance mutant is caught by nothing, because no remaining row sits exactly on the boundary. That is the whole argument for boundary rows, shown rather than asserted.
Derive every expected result from the policy paragraph and the inputs, on paper or in a separate file that never imports the grader.
Each mutant changes one operator, one default or one guard. A mutant that changes three things tells you nothing about which row is doing the work.
A surviving mutant is a gap in the table. Add the boundary row that kills it, or write down why the mutant is equivalent to the original and cannot be caught.
Tools such as mutmut automate this for a whole test suite. For a grader, a hand-written list like the one above is often clearer, because each mutant maps to one sentence of the policy.
Mutation testing proves your table can detect the defects you listed. It does not prove the policy is right, that the answer extraction upstream found the right number, or that an LLM judge agrees with people. Those need different measurements: adversarial answers and agreement with human labels.
Two kinds of mutation
Mutating the grader tests your tests. Mutating the answers tests the grader. Chapter 16 of the book does the second: it keeps every correct tool call and changes only the amount in the final explanation to 999,999 cents, and the harness reports that its named checks still pass. The chapter's response is to add an explicit acceptance status that says what was not graded, instead of letting a green passed field imply more than it checked. The same chapter measures a model grader against a person with Cohen's kappa, and shows a grader that always says "pass" agreeing with the person on 8 of 10 answers with a kappa of zero. Both habits belong in any harness whose number you plan to report.
Check the decision
What would show this approach wrong
The method has limits worth naming. If the policy is ambiguous, no boundary row recovers the missing decision. This policy says nothing about a truth of zero, where a relative tolerance of 1% allows no error at all; someone has to decide whether an absolute floor applies before you can write that row. If the person who wrote the grader also derived the expected values and made the same conceptual error in both, independence exists on paper only. And if every row passes under every mutant you can think of, either the grader is trivially simple or your mutants are too weak; try operators you did not write the table around.
Why this matters in a lab
Research results rest on graders: exact-match scorers, numeric tolerances, unit tests for code tasks, model judges. A grader bug does not throw an exception; it moves the number. Being able to say "our grader has a policy, a table derived from it and a mutation run showing each row can fail" is the difference between a result a reviewer can trust and one they have to re-derive. An interviewer who asks "how do you know your eval is measuring what you think?" is asking for exactly this: an independent oracle, boundary cases and evidence that the tests can fail.
Chapter 16 attacks a grader with a wrong amount, measures grader agreement with Cohen's kappa, and builds a harness that says what it did and did not check.
