Check That Your Judge Prompt's Examples Agree With Its Rubric
An LLM judge prompt usually has a rubric and a few examples. When an example contradicts the rubric, the model has two versions of the job, and you can't predict which one it follows on the next item.
Say the rubric gives partial credit when the final answer is right but the units are missing, and one example awards full credit to exactly that case. OpenAI's prompt engineering guide describes few-shot examples as a way to steer the model toward a task, recommends showing a diverse range of inputs with the desired outputs, and recommends evals that measure prompt behavior as you iterate. A contradictory example steers too, in the wrong direction.
The check, as a protocol you can run.
- Audit. For each example in the prompt, write down the grade the rubric alone would give. Any mismatch is a bug in the rubric or in the example; decide which.
- Human agreement. Have two people grade 50 fresh items using only the rubric. As an example of the arithmetic: if they agree on 42, that is 0.84, with a 95% interval of 1.96 × √(0.84 × 0.16 / 50) ≈ ±0.10. Read every disagreement. One that comes from an unclear rubric is a decision to make, and no prompt edit will make it for you.
- Paired comparison. Fix the rubric and the example, then run the judge before and after on the same 50 items. Count the items whose grade changed and read each one. Because the items are the same, a change comes from the prompt, not from drawing a different sample.
- Freeze. Keep the accepted examples and their grades in a test set, so a later edit that changes an example's grade fails a check instead of passing unnoticed.
Nobody has run this protocol for this note; the 42-of-50 figure is only there to show the arithmetic. Your numbers will differ.
Chapter 7 of the book explains why a prompt's exact words and examples need testing, and Chapter 16 derives the paired comparison.
