Ask Codex to Trace a Generated Results Table to Its Source
The results table in your paper repo shows 71.4 where the logs say 74.1. Editing the table by hand fixes the PDF until the next regeneration, and until then the paper and the logs disagree. In a lab, a number that no command reproduces is a finding against you.
Trace before you edit. Ask Codex to find, without changing anything:
- the script that writes the table;
- the exact command that runs it;
- the log file and field that produce the wrong cell.
Then fix the source, whether that is a wrong split name in the aggregation or a stale run id, and regenerate.
The check. Regenerate a second time and run git diff --stat. The second run should produce no diff at all. Compared with the version you started from, only the intended cells should change. If other cells moved, the generator depends on something you did not pin, such as file ordering or an unseeded sample.
Make the rule stick in review. OpenAI's GitHub integration guide says Codex searches the repository for AGENTS.md files and follows the applicable code review rules, which go under a ## Code Review Rules heading. You request a review by commenting @codex review on the pull request. A rule like this makes the habit a check:
## Code Review Rules
- Flag any edit under tables/ that has no matching change to its
generator script or to the logs it reads.Keep the generator's own tests in the verification path. The question that matters is whether the fix survives regeneration, not whether the checked-in table looks right today. Chapter 20 measures whether a written report matches what actually ran.
