Give Claude a Failing Eval Fixture Before You Ask for a Fix

"The accuracy looks too high" gives Claude Code nothing it can check. A three-row fixture does.

Suppose the eval set was merged from two shards and some examples appear twice. The scorer averages over rows. Build the smallest case that shows it:

By rows, accuracy is 2/3 ≈ 0.67. By unique examples, it is 1/2 = 0.50. Duplicated examples count extra, and a real eval set can hide hundreds of them, which is how a score drifts up with no change to the model.

Anthropic's best-practices guide recommends this shape of request: give the symptom and the likely location, write a failing test that reproduces the issue, then fix it, and address the root cause rather than suppress the symptom. So ask:

In tests/test_scorer.py, add a test using this three-row fixture that expects
accuracy 0.50. Run it and show me it failing. Then fix score() in
eval/scorer.py to deduplicate by example_id, and run the test again.

You want to see the red run before the green one. A test that has never failed has not shown it can catch this bug.

Then add the case that keeps the fix honest. Two different ids with identical prompt text and different reference answers are two examples, not one. A fix that deduplicates by prompt text would wrongly merge them. Put that case in the same test file. It protects the rule, deduplicate by id, rather than whatever the new code happens to do.

Chapter 16 treats an eval score as a measurement, which starts with counting each case once.

Claude Code documentation.