Plan When a Change Can Move Your Eval Numbers

A one-line change to the dedup key between your train and test splits can move every number in the results table. A 200-line refactor of the plotting code usually moves none. Diff length is a poor guide to consequences, and in a research repo the consequence that matters most is a reported number changing without anyone deciding it should.

Use Plan Mode when a change touches the scorer, the data split, the prompt template, the sampling settings or anything else a reported number depends on. Start with claude --permission-mode plan, or press Shift+Tab mid-session until the status bar shows ⏸ plan mode on. Per the documented workflow, Claude reads files and proposes a plan, and makes no edits until you approve. A prompt that says "plan first" is not the same as selecting the mode.

Ask the plan for four things: the current behavior, the proposed change, the files and interfaces it touches, and which results should change and which must not. That last item is what makes the plan testable.

Then hold the change to its plan:

uv run python -m eval.run --subset fixed-200 --out before.csv
# approve the plan, let Claude implement it
uv run python -m eval.run --subset fixed-200 --out after.csv
diff before.csv after.csv

(The eval command is a placeholder; use your repo's own.) For a refactor, the expected result is an empty diff. For a scorer fix, every changed row should be one the plan predicted, and you should be able to explain each. Rows that changed without a prediction are the finding.

For a small, reversible edit that touches no reported number, a separate planning pass may add little. Keep the before-and-after diff either way. The book's evaluation chapter shows how to tell a real change in a score from noise once the numbers do move.