AI Judge Calibration: Read the Confusion Matrix First

The judge that looked fine on paper

Imagine a support system drafts replies to customer tickets, and a second model, the "judge," reads each draft and flags whether it contains an error. Four replies come back from a test run. One is correct. One recommends the wrong product entirely. One is polished, accurate, and recommends a competitor's product by name. One is ambiguous, because the customer's request could reasonably be read two ways. The judge flags the wrong-product reply. It flags nothing else. Against the three clear human labels, it gets two right: 67% accuracy. The ambiguous case stays outside that denominator and remains a separate review task. Would you let this judge decide, unattended, which drafts get sent and which get held for review?

For this hypothetical support policy, competitor recommendations require review before sending. The missed case therefore blocks automatic acceptance under that policy. Another business could reasonably allow competitor comparisons, or consider wrong-product advice more costly. Write the policy and the consequence of each mistake before grading the judge. This is the actual teaching case behind Prof Rod's course lesson "Evaluation Engineering Across Both Quadrants", where a fictional engineer named Sarah gets a technically correct reply recommending a competitor and nothing in her test suite catches it, because her checks were built for a different kind of failure. That scenario is a teaching illustration, not a report of a measured incident, but the structural problem it demonstrates is real and worth working through with actual numbers.

What AI judge calibration measures

An AI judge, in this context, is a second model given a transcript or a draft and asked to output a verdict: did this response contain an error, and if so, what kind. It is attractive because it scales. A human reviewing every ticket does not scale past a certain volume; a judge can reduce the number of cases that require a full human reading. But a judge is only useful to the extent that its verdicts track what a human reviewer, given the same case, would have said. That tracking has to be measured directly, case by case, not assumed from a headline accuracy number.

LangChain's tuned-evaluator announcement describes a managed evaluator for perceived error in eligible traces. Whatever performance that evaluator reports in its own testing describes its own evaluation set, on its own criteria. It does not describe your ticket mix, your error taxonomy, or which mistake costs you the most. Anthropic's agent-evaluation guidance distinguishes tasks, graders, trajectories and outcomes and discusses human calibration. A judge tuned well on someone else's traces is not evidence it is tuned well on yours. The only way to know is to build a small labeled set of your own cases and check the judge against it, blind, before trusting a single verdict it produces on a case you haven't already labeled by hand.

Working the four-case example all the way through

Here is the worked example in full, with every number labeled as part of a hypothetical fixture, not a measured production run. Four authored customer-service replies, with human labels written down before anyone looks at what the judge says:

The judge caught case 2, the obvious wrong-product mistake. It missed case 3, the competitor recommendation, calling it clean. Case 4 stays labeled ambiguous by the human reviewer rather than forced into a binary, because pretending an ambiguous case has a clean ground truth manufactures a false confidence in whatever number comes out next.

Restricting the calculation to the three clearly labeled cases, the judge's confusion matrix looks like this:

From this small table: precision on the "error" call is 1 out of 1, or 100%, because the one time the judge flagged an error, it was right. Recall is 1 out of 2, or 50%, because of the two actual errors in the set, the judge caught only one. Overall accuracy across the three non-ambiguous cases is 2 out of 3, roughly 67%, treating the correct call on case 1 and the correct call on case 2 as two of three right answers. None of these numbers, on their own, tell you whether to trust the judge. What tells you is which case sits in the missed half of that recall figure, and whether that case belongs to your highest-cost error family.

Keep the counts beside the percentages. The 100% precision result rests on a single flag. If the next flagged reply is legitimate, precision drops to 50%. That sensitivity is visible immediately from the counts and easy to miss in a dashboard displaying only a rounded percentage.

What accuracy hides and why the family matters more than the count

Take the numbers above and imagine a fifth, sixth, and seventh case, all easy wrong-product mistakes the judge catches cleanly. With those three additional correct verdicts, accuracy becomes five out of six, about 83%. Adding more correctly graded examples can keep raising aggregate accuracy while leaving the same important miss unresolved. Nothing about that climbing number changes the fact that the judge still misses competitor recommendations. A team watching only the aggregate number would see it improve release over release and conclude the judge is getting more trustworthy, right up until a competitor-recommendation case slips through into a customer's inbox.

Define a separate acceptance rule for each costly error family. The rule needs both a metric and a sample-size requirement. A hypothetical 95% recall threshold means tolerating up to five misses in every hundred true errors in the evaluation set; it does not mean that 95% of all replies are safe. Decide whether that miss rate is acceptable using the consequences of an error and the capacity of the review queue. One successful example out of one is too little evidence to establish a dependable 100% recall rate.

Running the calibration check as an ordered procedure

The four-case table above is a toy illustration sized for explanation, held in a Python list or a small in-memory table, not a production labeling pipeline with concurrent writers, audit trails, or storage guarantees. A real calibration run needs enough examples per family to estimate the required performance with useful uncertainty bounds. Store the examples durably and use more than one reviewer where cases are contentious; the appropriate sample size depends on the tolerated miss rate. The mechanism, though, is identical at any scale:

What would prove this judge unfit and what would prove it fit

Write the promotion rule before inspecting a held-out set. The course uses an 85% agreement teaching threshold; agreement is the fraction of matching human and judge labels. Recall uses a different denominator: actual errors. Borrowing that 85% figure as a recall threshold would silently change the rule. Set the required recall, precision and uncertainty bounds from the intended routing decision instead. The four cases here are enough to expose a missed requirement, but cannot establish readiness for unattended use.

The falsifying test is concrete: freeze the candidate judge's prompt, inject one competitor-recommendation case it is known to miss and one legitimate competitor comparison it must correctly leave unflagged, then compare its verdicts against fresh blind human labels on both. If the judge still misses the case it should catch, or flags the legitimate comparison as an error, that is a fail on this fixture, not a pass with an asterisk. Changing only the judge's prompt while holding the four original cases fixed and checking which family's verdict shifts is a useful diagnostic here too, since a prompt edit that fixes competitor detection but silently degrades wrong-product detection has traded one blind spot for another rather than closing the gap.

Check the decision

Handing calibration off to whoever owns the review queue

A calibration result only protects a team if someone is accountable for acting on it. That means naming, in writing, who owns each error family's human labels, which family is treated as business-critical enough to block a judge's promotion to auto-approval, what routing action follows a below-threshold family, and who has authority to change the acceptance threshold later. Without that handover record, a calibration exercise becomes a one-time report that nobody revisits, and the judge drifts back toward being trusted on its aggregate score simply because no one is watching the family-level breakdown anymore. Whoever owns the regression set should hold back cases from each important family that the prompt author has not seen during tuning. A whole unseen family can be an additional stress test, but it cannot replace held-out coverage of the requirements being approved. None of this requires exotic tooling. It requires a table like the one above, a habit of relabeling by hand before checking the judge, and a standing rule that no single accuracy percentage authorizes automatic acceptance on its own.