Count Every Row Before You Trust a Dataset Pull
You ask Claude Code to pull last month's eval runs from your tracker through an MCP server, or to page through an API for a dataset. The file it writes is valid JSON, neatly formatted, and holds only the first page.
Nothing fails loudly when pagination stops early. You get a smaller n, and every number computed from it looks fine.
One Claude Code detail is worth knowing here. Per the MCP documentation, Claude Code shows a warning when an MCP tool's output exceeds 10,000 tokens and, by default, allows at most 25,000, a limit you can change with MAX_MCP_OUTPUT_TOKENS. When a text result is over the limit, Claude Code saves it to a file and puts a message naming that file in the conversation. So a large result may sit in a file the agent has to open, not in its context. Ask where the data came from before you ask what it says.
The check. Before anything downstream runs, reconcile the pull:
jq -s 'length' runs.jsonl # rows you received
jq -r '.id' runs.jsonl | sort | uniq -d | wc -l # duplicate ids, expect 0Compare the first number with a count you get another way: the tracker's own total, the eval set's manifest, or the number of runs you launched. They should match exactly.
Then test the pagination itself with a synthetic source that returns three pages, the last one partial, and fails once on the middle page. Expected: the pull retries or reports itself incomplete. It must not write a shorter file that looks finished, and a retry must not duplicate the rows it already has. Exactly one order: lost replies, retries and idempotency works through retries that neither lose nor repeat a record.
A mismatch here takes a minute to find now and a week to explain after a results table has circulated.
