Write Review Rules Around Your Experiment's Invariants
"Review carefully" gives a code reviewer nothing to prioritize. An invariant does. In research code, the invariants are the rules that keep a number honest:
- no eval item appears in the few-shot pool;
- items that failed to parse stay in the accuracy denominator;
- every run records its seed and model id.
Break the second one and nothing crashes. Suppose 170 answers are correct, 20 fail to parse, and 210 items were run. Honest accuracy is 170/210 ≈ 81%. Drop the parse failures from the denominator and you report 170/190 ≈ 89%, a gain of more than eight points that no model produced.
For Codex code review in GitHub, OpenAI's AGENTS.md documentation says to add a ## Code Review Rules section "to the AGENTS.md closest to the code the rules govern." It asks you to "keep rules concise, explain the behavior to flag and any safe path or exception, and reserve formatting and lint checks for CI." That format suits invariants: each rule names one behavior and the cases where it may differ.
The check. Add the three rules above to the AGENTS.md beside your eval code. Then test the rules with two pull requests on a scratch branch:
- A change that quietly filters parse failures out before computing accuracy. Expected: a finding that cites the rule and the line.
- A harmless refactor of the same file, such as renaming a variable. Expected: no finding.
The second test matters as much as the first. A rule that fires on every change to the file trains everyone to ignore it, and then it misses the change that counts. A whole day: reliability, honest reports and readiness covers the report these invariants protect.
