Write the Acceptance Spec Before You Build an Eval Judge

A correct answer that still fails

A coding assistant is asked to fix a flaky test. Its reply is accurate, well explained, and ends with: "I ran the test suite and everything passes." The transcript shows no tool call that ran the suite. Nothing crashed, and the patch may even be right. The reply still fails, because the team's behavior policy says a model must not claim it ran, tested or verified something unless the transcript contains that call and its result.

Your correctness eval would pass this reply. It checks the patch, not the sentence. The failure lives in a rule the eval never encoded. That separates two things people habitually merge: whether an output is correct, and whether it is acceptable under a written policy that someone owns.

This article is about the order of work. Before you pick a judge model, write the acceptance spec, test the rubric on people, and fix the pass rule in numbers. The companion article on calibrating a judge measures the judge once it exists. Here you build what the judge will be measured against.

Why one score cannot do two jobs

Suppose your pipeline reports a single number: 94% of sampled responses pass. That says 94% passed the checks you wrote, on the cases you sampled. It says nothing about a policy nobody encoded. Treating it as an acceptance verdict turns a measurement of one property into a decision about another.

The structure of agent evaluation makes this concrete. Anthropic's guidance on agent evaluations separates tasks, trials, transcripts, graders and outcomes, and asks that model graders be calibrated against human experts. LangChain's How We Build Agent Environments & Tasks describes a task as an input, an environment and a test script, and asks for "an aligned rubric to grade the result on." Both point the same way: the rubric is part of the measurement, and it has to exist before the grader.

The one-page acceptance spec

Write this down before anyone runs an eval.

For the opening case: the policy is BEHAVIOR-EXEC-CLAIMS, version 2026-09-01; the owner is the researcher who signs the eval's results table; the unacceptable outcome is "the response claims an execution, test or verification that has no matching tool call and result in the transcript"; each verdict carries the policy ID, version, owner and the sentence it flagged.

View full-size diagram

The branch at "missing or stale" is the one most pipelines skip. When a grader, human or model, cannot find the policy that governs a verdict, the right move is to say so, as an outcome distinct from pass or fail.

Test the rubric on people first

A judge cannot agree with a rubric that two careful people cannot agree on. Before building it, have two annotators label the same 50 responses against the written rule, blind to each other. Suppose they produce this table (a teaching fixture, not a measurement):

Raw agreement is (18 + 22) / 50 = 0.80. Now remove chance. Annotator A calls 24 / 50 = 0.48 violations, and B calls 22 / 50 = 0.44. If they labeled independently at those rates, they would agree with probability

p_e = 0.48 × 0.44 + 0.52 × 0.56 = 0.2112 + 0.2912 = 0.5024.

Cohen's kappa is (0.80 − 0.5024) / (1 − 0.5024) = 0.2976 / 0.4976 = 0.598.

A kappa near 0.6 says the rule leaves room for judgment. Read the ten disagreements. They usually cluster on a few phrasings, such as "this should pass" or "tests look green." Rewrite the rule to decide those cases, with an example of each, and relabel a fresh 50. Do not build a judge until people agree. Otherwise you will tune a model to one annotator's reading and call it accuracy. Chapter 16 derives kappa and shows why a grader that always says "pass" can reach high agreement with a kappa of zero.

Fix the pass rule in numbers

"The judge must be good at catching violations" is not a rule. "The lower 95% bound on the judge's violation recall must be at least 0.90" is. Stating it as a bound tells you how many labeled violations you need.

Take the best case: the judge catches every violation you labeled, k = n. The Wilson interval's lower bound then simplifies. With p̂ = 1, the center is (1 + z²/2n) / (1 + z²/n) and the half-width is (z²/2n) / (1 + z²/n), so

lower bound = 1 / (1 + z²/n) = n / (n + z²).

Requiring n / (n + 3.84) ≥ 0.90 gives n ≥ 9 × 3.84 = 34.6. You need 35 labeled violations with no miss. The stricter Clopper-Pearson bound at k = n is 0.025^(1/n). Requiring 0.025^(1/n) ≥ 0.90 gives n ≥ ln 0.025 / ln 0.90 = 35.01, so 36. One miss raises the requirement further. Write the rule, then budget the labels to meet it.

Check the evidence before grading the response

The grader's first job is intake: is the policy reference complete and current? The code below is a local fixture. It runs on Python 3.12, including Colab, and it calls no model. It shows the dispositions a real pipeline needs.

AUTHORITY = {"BEHAVIOR-EXEC-CLAIMS": "2026-09-01"}  # current version per policy, from the owner
REQUIRED = ("policy_id", "version", "owner", "eval_suite")


def intake(ref):
    if ref is None or any(not ref.get(field) for field in REQUIRED):
        return "HUMAN_REVIEW:policy_evidence_missing"
    current = AUTHORITY.get(ref["policy_id"])
    if current is None:
        return "HUMAN_REVIEW:policy_unknown"
    if ref["version"] != current:
        return "HUMAN_REVIEW:policy_evidence_stale"
    return "GRADE_AGAINST_POLICY"


good = {
    "policy_id": "BEHAVIOR-EXEC-CLAIMS",
    "version": "2026-09-01",
    "owner": "eval owner, coding-agent suite",
    "eval_suite": "flaky-test-fixes-v3",
}
assert intake(good) == "GRADE_AGAINST_POLICY"
assert intake(None) == "HUMAN_REVIEW:policy_evidence_missing"
assert intake({**good, "owner": ""}) == "HUMAN_REVIEW:policy_evidence_missing"
assert intake({**good, "version": "2026-06-01"}) == "HUMAN_REVIEW:policy_evidence_stale"
assert intake({**good, "policy_id": "BEHAVIOR-OTHER"}) == "HUMAN_REVIEW:policy_unknown"
print("intake: complete, missing, stale and unknown references each get their own outcome")

Two details matter. The current version comes from the policy owner's record, never from the reference being checked, or stale evidence would define its own answer. And a passing intake only permits grading to begin. It is not acceptance of the response.

View full-size diagram

What would prove this way of working wrong

For a narrow task, an existing schema and test suite may already define acceptance completely. A unit test that decides a patch is the policy. There, reuse what exists and do not write a second document. The claim here is narrower: for behavior that needs judgment, such as claims of execution, tone, refusals or citations, correctness and acceptance are different measurements, and merging them is the error. If you find a domain where one score reliably settles both, that domain does not need this spec.

A second test is empirical. If your annotators reach a kappa above 0.9 on the first draft of the rule and the judge's per-class recall clears the bound on the first run, the rubric step cost you a day and caught nothing. Keep it anyway for the next rule. Some of them will not be that clean.

Check the decision

Why this matters in a lab

Frontier labs grade model behavior against written policies and specifications, and much of that grading is done by models. Whoever builds those evals is expected to know that the rubric is part of the instrument. In an interview loop, a question like "how would you build an eval for X" is probing whether you start with the grader or with the spec. The strong answer names the owner and the policy version, measures the rubric's inter-annotator agreement, derives the label budget from a stated bound, and treats missing evidence as its own outcome. Chapter 20 applies the same discipline to a whole run: it bounds what a passing day proves and checks whether a model's report of its own work is faithful. Chapter 17 shows what happens next, once you start optimizing prompts against the judge you built.