Turn a Notebook Cell Into a Contract Your Eval Harness Can Call

The cell that works is not the function that ships

You have a notebook cell that takes one judge verdict, checks it and prints a tidy row for your results table. You ran it on three verdicts you typed yourself, and it worked every time. A teammate wants to call it from the eval harness, where verdicts arrive as JSON from a queue, thousands per run. They ask one question: what happens when the judge sends the score as "3" instead of 3?

You do not know, because you never sent that value. The cell was tested against inputs you chose, not the inputs a model and a transport layer will actually produce.

That gap, between "ran once on data I built" and "a function another program can depend on", is the subject here. A contract is not a nicer word for a function. It is a declared boundary: which inputs are accepted, which are rejected and why, and what the caller receives either way. Until those are written down and checked, a notebook result is a demonstration.

What a contract has to name

Accepted inputs. Field names, types and ranges the function processes without complaint. A rubric score typed as an integer from 0 to 4 is a commitment. If the sender provides 5, the function refuses before doing any work; it does not clamp the value to 4.

Rejected inputs, with a stable reason. A caller needs to know not only that -1 fails but why, and to tell that apart from a missing field. Missing and null are different facts. Collapsing both into one "bad input" forces every caller to guess.

One output shape. On success, the same field names and types every time, whichever path produced them. If the local function returns an integer score and the queue consumer returns the string "3", you have two contracts wearing one name.

A worked contract: validate_verdict

The notebook version, made strict. It runs on Python 3.12 (Colab's version) with Pydantic 2.

import json
from typing import Union

from pydantic import BaseModel, ConfigDict, Field, ValidationError


class Contract(BaseModel):
    model_config = ConfigDict(strict=True, extra="forbid")


class Verdict(Contract):
    case_id: str = Field(min_length=1, max_length=40)
    score: int = Field(ge=0, le=4)


class RejectedVerdict(BaseModel):
    error_code: str
    field: str
    detail: str


def validate_verdict(payload: dict) -> Union[Verdict, RejectedVerdict]:
    try:
        return Verdict.model_validate(payload)
    except ValidationError as exc:
        first = exc.errors(include_input=False, include_context=False)[0]
        return RejectedVerdict(
            error_code=first["type"],
            field=".".join(str(part) for part in first["loc"]),
            detail="case_id or score failed validation",
        )

Call it with {"case_id": "q-0417", "score": 3} and you get a Verdict with an integer score. Call it with "score": "3" and you get a RejectedVerdict naming score and the code int_type, because strict mode does not convert a numeric string. That refusal is a decision worth stating: the judge is expected to produce the declared type, and a judge that does not is telling you something about its reliability. A companion article shows what lax parsing does to a pass rate.

The function returns one of exactly two typed shapes, never a bare dictionary. Programming errors can still raise. Pin the Pydantic version and test the error codes before treating them as a permanent interface.

Where the contract breaks on the way to the harness

The notebook never exercised the path your teammate will use: JSON arriving from somewhere else. Here is a toy in-process adapter that plays that role. It is not a claim about any deployed queue or server.

def request_adapter(raw_body: str) -> str:
    """Toy stand-in for a queue consumer or HTTP handler: decode JSON,
    call validate_verdict, encode the result. Runs in-process only."""
    try:
        payload = json.loads(raw_body)
    except json.JSONDecodeError:
        return RejectedVerdict(error_code="invalid_json", field="body",
                               detail="body must be valid JSON").model_dump_json()
    return validate_verdict(payload).model_dump_json()


print(request_adapter('{"case_id": "q-0417", "score": 3}'))
print(request_adapter('{"case_id": "q-0417", "score": "3"}'))

{"case_id":"q-0417","score":3}
{"error_code":"int_type","field":"score","detail":"case_id or score failed validation"}

The adapter and the direct call reach the same decision for the same body. That proves something narrow: within this process, both paths share one type decision. It proves nothing about a real message broker, a client library's encoder, or a framework that parses request bodies before your code sees them.

The failure this setup is built to catch is an adapter written separately from the validator, one that "helpfully" converts "3" to 3 while decoding. Then the validator never sees the string and cannot reject it. JSON does distinguish 3 from "3"; a hand-rolled converter can erase that difference without anyone noticing until the pass rate is off.

View full-size diagram

The risk sits on the arrow from "Adapter decodes body" to "validate_verdict checks type". The fix is not more validation code. It is making one function own the type decision and having every entry point, notebook or harness, call it with the untouched payload.

Build the expected-results table first

Before wiring any real transport, write the table every entry point must satisfy. The expected outcomes come from the contract, not from running the code.

Then run it through both paths. The codes are Pydantic v2 codes, confirmed on 2026-09-30 with Python 3.12.13 and Pydantic 2.13.5.

cases = [
    ({"case_id": "q-0417", "score": 3}, None),
    ({"case_id": "q-0417", "score": "3"}, "int_type"),
    ({"case_id": "q-0417", "score": True}, "int_type"),
    ({"case_id": "q-0417", "score": 3.0}, "int_type"),
    ({"case_id": "q-0417", "score": -1}, "greater_than_equal"),
    ({"case_id": "q-0417", "score": 5}, "less_than_equal"),
    ({"case_id": "q-0417"}, "missing"),
    ({"case_id": "q-0417", "score": None}, "int_type"),
]
for payload, expected_error in cases:
    direct = validate_verdict(payload).model_dump()
    adapted = json.loads(request_adapter(json.dumps(payload)))
    assert direct == adapted
    assert direct.get("error_code") == expected_error
assert json.loads(request_adapter("{"))["error_code"] == "invalid_json"
print("Eight typed cases agree; malformed JSON has a declared rejection")

Each row checks a declared outcome and agreement between the two paths. A disagreement on any row is a regression, however harmless it looks. A local function returning 3 while the harness path returns "3" is the kind of drift someone finds weeks later, when "3" * 2 is "33" and a mean score comes out as nonsense.

View full-size diagram

Where responsibility ends

Passing this contract establishes something specific. The verdict has the declared shape and type, in the declared range. It does not establish that q-0417 is a case in the current eval split, that the judge's 3 is the score a person would give, or that the verdict belongs to this run rather than last week's. Those are separate checks against separate evidence: the split manifest, a human-labeled agreement sample, a run identifier. Chapter 16 of the book measures judge agreement with Cohen's kappa and shows why raw agreement overstates it. Anthropic's notes on writing tools for agents make the matching design point: describe inputs, outputs and failures explicitly, because a caller relies on what is described.

Strictness is also narrower than it looks. Pydantic's strict-mode documentation notes that behavior differs by type and that JSON input has its own exceptions for types such as dates. The integer case here does not generalize to every field; check each type you add.

Check the decision

What would prove this wrong

If you ran the table against the real harness entry point, not the toy adapter, and found that the queue path turned "3" into 3 while the notebook function still rejected it, that would falsify the claim that one shared function guarantees the same behavior everywhere. Something between the judge and your function, a framework's body parser, a serializer or a proxy, would be rewriting the payload. That failure is common, and finding it is the reason to run the table against the entry point your callers actually use. Nothing here claims that check has been run against a real pipeline; it is the next piece of work.

Why this matters in a lab

Research code moves from notebook to pipeline constantly: a scoring cell becomes part of a harness, a data-cleaning cell becomes a preprocessing job. The habit being tested is whether you can make that move without changing the result. In a take-home work sample, a reviewer can see it at once: one function with a typed input and output, a table of rejected inputs with reasons, one test that runs the table through every entry point, and a sentence stating what the contract does not prove. That reads as someone whose numbers survive being run by other people.