Teach Claude That a Blank Eval Score Is Not a Zero
In an eval results table, a blank score can mean the run timed out, the API returned an error or the grader crashed. A zero means the model answered and was wrong. Merging the two changes the headline number, and a reviewer who finds out later will distrust every other number in the table.
The arithmetic. 200 items, 150 correct, 20 with no score. Count the missing as wrong and accuracy is 150 / 200 = 0.75. Drop them and it is 150 / 180 ≈ 0.83. That gap is larger than the 95% interval half-width at this sample size, 1.96 × √(0.75 × 0.25 / 200) ≈ 0.06. Neither figure is honest on its own. Report the scored count, the missing count and why each item is missing, and say which convention the headline uses.
Before asking Claude Code to write or change the aggregation, write down what each empty value means in each column. Then give it a fixture and an expected result you computed by hand:
item_id,score,status
1,1,ok
2,0,ok
3,,timeout
4,NaN,grader_error
5,error,api_errorExpected: 2 items scored, 1 correct, accuracy 0.50 over scored items, 3 missing, with their reasons listed. Watch the defaults: pandas reads the blank and NaN as missing values, and mean() skips missing values unless told otherwise, so a naive average silently reports accuracy over scored items only. The string error makes the whole column non-numeric, and the usual fix, pd.to_numeric(..., errors="coerce"), turns it into one more silent missing value. On this fixture that path gives a mean of 0.5 over 2 items and says nothing about the other 3.
Claude Code's best practices say to give Claude a check it can run, such as a script that compares output against a fixture. This fixture and its expected result are that check; ask Claude to add them as a test.
Then try a run in which every score is missing. The report should say there are no scored items, not "accuracy 0.0." The book's acceptance chapter covers writing reports that say plainly what was and was not measured.
