Check a Codex Fix Against the Original Failing Case

Your eval harness scores cases with an empty reference answer as correct, which inflates accuracy by the share of such cases. A fix arrives with a new test, and the test passes. That shows the fix and its own test agree. It does not show the original failing case is fixed, and a research team will ask which one you checked before you rerun the leaderboard.

Make the original failure a test, and commit it before the fix:

git log --oneline -2                       # the repro test commit, then the fix commit
git checkout HEAD~1 && uv run pytest tests/test_empty_reference.py   # expect: fails
git checkout - && uv run pytest tests/test_empty_reference.py        # expect: passes

Then check one neighbor that must not change, such as a case with a normal reference that was scored correctly before. If the fix moves the headline number, say by how much and why: with 40 empty-reference cases in 1,000, accuracy can drop by up to 0.04.

Codex can add a review pass on GitHub. The integration documentation says to comment @codex review on the pull request; Codex reviews the diff, follows the repository's AGENTS.md guidance, and in GitHub flags only P0 and P1 issues. So a quiet review is not evidence that the fix addresses the reported failure. The same page describes a ## Code Review Rules section in the AGENTS.md closest to the code, and recommends leaving mechanical checks to CI. A rule like this fits:

## Code Review Rules
- A bug-fix PR must add a test that reproduces the reported failure and name the issue it closes.

If your local environment differs from the one that produced the report, say so rather than treating a local pass as the final word. The book's acceptance chapter covers writing an honest report of what was and was not checked.

OpenAI documentation.