Test the Boundary Your Mock Removes From an Eval Harness
A mocked model client returns exactly what you told it to: well-formed JSON, every field present, no errors. The real API also returns responses cut off at the token limit, refusals and rate-limit errors. If your eval harness has only been tested against the mock, it has never seen any of them.
The damage in research code is quiet. A truncated response fails to parse, the harness scores it as a wrong answer, and accuracy drops by an amount that has nothing to do with the model's ability. Or the harness skips it, and n shrinks without anyone noticing. Either way, the number in your results table measures your parser as much as the model.
When Claude Code proposes a test for the harness, ask which boundary the mock replaces and what the test still exercises. The Claude Code best-practices guide gives a well-scoped test request as an example, and it ends with two words: "avoid mocks." You do not have to drop doubles entirely. Keep one fast test with a double, and add tests that feed recorded real responses through the same parser.
The check. Take one recorded response from a real run, cut it in half, and pass it to your parser. Then pass a recorded refusal. Expected result: each lands in its own category in the results file, such as parse_error and refusal, and your results table reports those counts beside accuracy. If either one is silently scored as incorrect or dropped, you have found the gap.
Write the expected categories into the fixture by hand. A fixture whose expected values come from the code under test only proves that the code agrees with itself. Structured output and typed tools covers what a schema does and does not guarantee about a response.
