Measure What a Second Model Catches Before Adding It to Review
Measuring one reviewer means its recall on planted bugs and its precision on real findings. This note asks a different question: does a second reviewer catch what the first one misses? Adding a second model family to review a change in your eval harness sounds like diversity. Whether it is diversity is a number you can measure, and a research team will expect you to measure it before you make the second call routine.
The protocol. Copy the repository and seed bugs you know about: an off-by-one in a pass@k loop, a train/test split that leaks one shard, a metric averaged over rows instead of unique examples. Give reviewer A and reviewer B the same diff, the same spec and the same tests. Neither sees the other's findings. For each seeded bug, record caught or missed for each reviewer, and ask each one for an input that reproduces the failure, so a "catch" means something you can run.
The arithmetic. Suppose A catches 70% of the seeded bugs and B catches 60%. If their misses were independent, both would miss 0.3 × 0.4 = 0.12 of them, and the pair would catch 88%. Now count the bugs both actually missed. If that fraction sits near 0.12, B adds a different view. If it sits near 0.30, B mostly repeats A, and a different brand name did not buy independence.
The sample size. With 20 seeded bugs, a 70% catch rate has a 95% interval of about 1.96 × √(0.7 × 0.3 / 20) ≈ ±0.20, which is too wide to separate the two cases. Seed 50 or more. Also run both reviewers on a clean diff and count the false findings, because a reviewer that flags everything catches every bug.
Chapter 16 derives the interval and the paired comparison you need here. Agreement between two models is still two model opinions; the seeded bugs and the tests are the evidence.
