Ask Claude to Run the Data Prep Twice Before You Trust the Row Count
After a crash, someone reruns the eval-set builder. If the script appends instead of replacing, the test set now holds every example twice, and nothing fails.
That quietly changes your error bars. A 95% interval on an accuracy p measured over n examples has half-width 1.96 × √(p(1 − p)/n). Double n with copies and the interval narrows by 1/√2 ≈ 0.71, with no new evidence behind it. A reviewer comparing two models would read that as a sharper result than you have.
Anthropic's best-practices guide says to give Claude a check it can run, such as a script that diffs output against a fixture. Here the check is a second run. On a 50-row fixture, ask Claude Code:
Run scripts/build_eval_set.py on fixtures/small/ twice into out/.
After each run, print sha256sum for every file in out/, the row count,
and the number of unique example_id values. Report any difference.After the second run, all three should match the first: same hashes, same row count, unique ids equal to rows. If the row count doubled, the script appends. If the hashes changed but the counts did not, look for a shuffle without a fixed seed.
Some differences are legitimate, such as a manifest that records a timestamp. Name those files and exclude them by path rather than ignoring the whole diff. Then keep the check in the repo as a test, so the next edit to the pipeline has to pass it.
Chapter 12 derives why a repeated operation has to be idempotent when you cannot tell whether the first attempt finished.
