Stop Being the Agent's Verification Loop
You ask if the change is done. The agent says yes. You rerun the eval, and it isn't.
The Claude Code best-practices guide names the position you are in: without a check Claude can run, "you become the verification loop: every mistake waits for you to notice it."
The fix is a check that returns pass or fail and that the agent can run itself: a test suite, a build exit code, a linter, or a script that diffs output against a fixture. For research code, the fixture diff is the most useful and the least used. A change to scoring should reproduce a known number on a small fixed set before anyone runs it at scale.
The check. Commit a 10-item fixture with hand-checked expected scores, and a one-line comparison:
uv run python score.py fixtures/ten-items.jsonl | diff - fixtures/ten-items.expectedTell the agent to run it after every change and to keep going until the diff is empty. Then ask for evidence, not a verdict: the command it ran and what it printed. The guide makes the same request, "Have Claude show evidence rather than asserting success," and gives the reason: reviewing evidence is faster than rerunning the check yourself.
Know what the check cannot see. An empty diff on ten items says the scorer agrees with ten answers you checked by hand. It says nothing about item eleven, about noise at the scale of the full set, or about whether you are measuring the right thing. Write those limits into the report next to the result. The agent loop builds a loop that stops on a check like this one.
