Measure Human Agreement Before You Trust an LLM Judge
An LLM judge cannot give a stable quality signal if the people who wrote the rubric disagree about which model outputs should pass. OpenAI's evaluation guide lists disagreement among human experts as a challenge of human evals, and says to validate a judge's agreement against human labels and scale it once it consistently agrees with human annotations. Building and defending an eval like this is core work for a research-engineering role.
Step 1: measure the humans. Two reviewers grade the same 60 outputs pass or fail against one written rubric, without seeing each other's grades. Compute observed agreement p_o and Cohen's kappa, κ = (p_o − p_e) / (1 − p_e), where p_e is the agreement expected by chance from each reviewer's pass rate. A worked example of the arithmetic, not a measurement: if the reviewers agree on 51 of 60 (p_o = 0.85) and pass 42 and 39 items (70% and 65%), then p_e = 0.70 × 0.65 + 0.30 × 0.35 = 0.56 and κ = (0.85 − 0.56) / 0.44 ≈ 0.66.
Step 2: fix the rubric, not the reviewers. Read every disagreement and find the criterion behind it. The guide suggests examples at each score level and an explicit pass/fail threshold. Rewrite, then repeat on fresh items.
Step 3: test the judge on held-out items. Tune the judge prompt on half the labeled items and measure its agreement with the human consensus on the other half. If it agrees on 26 of 30, that is 0.87 ± 1.96 × √(0.87 × 0.13 / 30) ≈ ±0.12. A judge tuned and tested on the same items tells you little.
Step 4: probe the known biases. The guide names position bias and verbosity bias. For a pairwise judge, swap the two responses and count how often the verdict flips.
This site has not run this protocol on a particular task. The course repo's evaluation statistics exercise and the book's evaluation chapter cover the intervals.
