Test Your Results Scripts on Awkward Run Names
A results script that aggregates runs/*.jsonl works on clean names. Sweeps don't always produce clean names: lr 3e-4.jsonl, a file that starts with a dash, a quote in a config label.
If the script splits on spaces, or passes a leading-dash name to a command that reads it as an option, it skips or misreads those runs. The table still looks complete. The mean moves, n shrinks, and nothing tells you. In an eval, a silently dropped run is worse than a crash, because the number you report is now computed on a different sample than the one you describe.
Anthropic's best practices for Claude Code say to give Claude a check it can run, such as a script that diffs output against a fixture, and to ask for a failing test that reproduces the issue before the fix.
The check. Build the fixture yourself so you know the answer:
mkdir -p fixtures/runs
printf '{"score": 1}\n' > "fixtures/runs/lr 3e-4.jsonl"
printf '{"score": 0}\n' > "fixtures/runs/-seed1.jsonl"
printf '{"score": 1}\n' > "fixtures/runs/it's.jsonl"
printf '{"score": 1}\n' > "fixtures/runs/plain.jsonl"Then ask Claude Code for a test that runs the aggregation on this folder and asserts 4 rows and a mean of exactly 0.75. Expect the first version to fail on at least one name; that failure is the test earning its place. Inspect the output table yourself, not only the exit status.
Add two more cases: an empty runs folder and a missing one. The right behavior for both is a clear error, not a table with zero rows or a mean of NaN.
Why a smaller n matters, and how it widens the interval on your number, is in Chapter 16 of the book.
