Trace One Failed Eval Item Through the Whole Harness
Your eval run finished, the score dropped four points, and the log has a hundred warnings. The loudest one is rarely the cause. Start with one item that was scored wrong and follow it through every stage.
An eval harness is a pipeline: load the item, build the prompt, call the model, parse the response, score the answer, write the row. A failure can enter at any stage, and only one of those stages is the model. A prompt template that drops a field, a parser that reads the first number in the response, or a scorer that compares strings with different whitespace all show up in the table as model error.
The check. Pick one item id that the run scored incorrect. Pull every log line for it into one file and hand it to Claude Code. The best-practices guide documents piping data straight in:
grep '"item_id": 417' runs/latest/*.jsonl > item-417.log
cat item-417.log | claudeAsk for a timeline with one row per stage: what went in, what came out, and the file and line that produced it. Mark any stage where the item id disappears. Expected result: either a stage whose output stops matching what the next stage expects, or confirmation that the model's raw answer was wrong. Both are worth knowing before you spend a day on the model.
If the id disappears between two stages, the fix is instrumentation. Log the item id at every boundary, then rerun that one item. Repeat the trace for four more failures. If three of five trace to the harness rather than the model, your score is measuring your pipeline, and the four-point drop may not be about the model at all.
Keep API keys and private data out of any log you paste. Evaluation as measurement treats the harness as part of the instrument whose error you report.
