Measure a Model Reviewer's Recall and Precision Before You Trust It
Claude Code ships /code-review as a bundled skill. The commands reference says it reviews "the current diff, or a PR number, branch, or path you pass, for correctness bugs." The best-practices guide recommends the same split by hand: a second session reviews the change in a fresh context, without the reasoning that produced it. OpenAI's GitHub integration reviews a pull request when you comment @codex review, and in GitHub it flags only P0 and P1 issues.
Each one hands you findings, and none tells you how many to trust. Treat the reviewer as a detector and measure two numbers: how many known bugs it catches, and how many of its findings hold up.
A finding counts when a test fails. Ask the reviewer for a concrete failing input, then write the smallest test that fails on the revision under review. "The loader drops the last partial batch" becomes:
def test_loader_keeps_partial_batch():
preds = run_eval(list(range(10)), batch_size=4)
assert len(preds) == 10If it fails, fix the code and keep the test as a regression guard. If it passes, reply on the thread with the test. Either way the next reviewer, human or model, inherits a fact instead of an argument.
Recall, on bugs you planted. Research code has bugs that change numbers without raising errors: a split that shuffles before it separates train from test, an accuracy denominator that drops parse failures, a seed set after the data loader starts. On a scratch branch, plant one per trial in an otherwise clean diff, write down what a correct finding would say, run the reviewer and record hit or miss. Seven caught out of ten is 70%, with a 95% interval of 1.96 × √(0.7 × 0.3 / 10) ≈ ±28 points. Thirty trials bring it to about ±16.
Precision, on findings you triage. The best-practices guide warns that "a reviewer prompted to find gaps will usually report some, even when the work is sound," and advises asking it to flag only gaps that affect correctness or the stated requirements. To see how much noise is left, run it on diffs you know are sound, such as a pure rename or a refactor with identical results on a fixed eval subset. Every finding there is false by construction. On real diffs, sort each finding into confirmed (a test fails), rejected (you showed the state cannot occur, and wrote down why) or open. Precision is confirmed over reported. If 12 of 20 findings reproduce, that is 0.60 ± 0.21, too wide to tell a good reviewer from a coin flip. The interval tells you how many findings you need before the number means something.
The numbers here are arithmetic, not measurements of any reviewer. The misses are the useful output: they tell you which invariants to state in your review instructions and which to guard with a test.
Fresh context removes the writer's argument from the review. It does not remove the model's own habits, and adding reviewers that vote does not fix a premise they share. Whether a second model family adds anything is its own measurement: compare the two reviewers' joint misses. A bug hunt also misses scope; for that, find the change you never asked for. Chapter 16 derives the interval and why it shrinks with √n.
