Let Claude Interview You Before You Build an Eval
"Can you build an eval for tool-use regressions?" sounds like one task. It hides a dozen decisions: which model and snapshot, what counts as a pass, how many items, what a false alarm costs, what is out of scope.
Anthropic's best practices for Claude Code suggest starting a larger task by having Claude interview you with the AskUserQuestion tool, about implementation, edge cases, concerns and tradeoffs, until it can write a complete spec to SPEC.md. Then start a fresh session to implement it, with clean context and the spec to refer to. The docs add that the most useful specs name the files and interfaces involved, state what is out of scope, and end with an end-to-end verification step.
For an eval, push the interview toward numbers. Ask Claude to include this question: how small a regression must the eval detect? The answer sets n. A pass rate near 0.8 measured on 200 items has a 95% interval of 1.96 × √(0.8 × 0.2 / 200) ≈ ±0.055. A 2-point drop is invisible at that size; detecting it needs a paired design or far more items.
The interview can find a question you missed. It cannot answer for the person who owns the decision. A plausible default the model proposes is still an assumption until someone who owns the outcome confirms it.
The check. Before implementation starts, read SPEC.md and count three things: open assumptions, each with a named owner; an explicit out-of-scope list; and a final step that runs the eval end to end on a small known set with a known expected score. If any count is zero, the spec is not ready.
The same habit serves a take-home work sample. When the prompt leaves something unspecified, a short written list of the assumptions you made shows the reviewer how you reasoned.
Chapter 16 of the book derives the interval above and the paired comparison that detects small changes.
