Why Schema-Valid Model Output Can Still Be Wrong, and How to Check It

Teaching fixture: The eval sets, token counts and prices below are invented for arithmetic you can check. The prices are not a quote from any provider. Money is USD, stored as integer cents.

The plan validates. It also reads the test split.

An agent in your research repo is asked to plan the next evaluation of a prompt change. It returns JSON:

{"eval_set": "reasoning-v3", "split": "test", "n_cases": 400}

Every field has the right type. The eval set exists. The split is one of the allowed strings, "dev" or "test". The case count is a positive integer under the cap. A schema that says all of that passes this object.

It is still the wrong plan, twice over. You are iterating on a prompt, so the policy says you may only look at the dev split; the 1,000-case test split is frozen for the final report, and every peek at it during iteration spends some of its value as an unbiased estimate. And 400 is the size of the dev split, not the test split, so the number only looks right because the model copied it from the wrong row.

Two questions sound alike and are not: "Is this JSON shaped correctly?" and "Is this the correct research decision?" A parser and a schema answer the first. The second needs facts the model never controls: the eval registry, the split rule, the sampling policy, the price table and the budget. This article puts that second check in code, then measures how often the first check passes while the second one fails.

How often schema-valid means wrong

You do not have to guess at this gap. Chapter 2 of the book ran the experiment: two small open models turned twelve plain-language requests into {"sku", "quantity"}, five samples each, once with JSON asked for in the prompt and once with the schema enforced by constrained decoding. With the schema enforced, both models produced 60 of 60 answers that parsed and fit the schema. The correctness column is the one to read:

For the smaller model, the valid-but-wrong rate is p = 16/60 = 0.267. A normal-approximation 95% interval is p ± 1.96·√(p(1−p)/n):

• p(1−p)/n = 0.267 × 0.733 / 60 = 0.00326;
• √0.00326 = 0.0571;
• 1.96 × 0.0571 = 0.112.

That gives 26.7% ± 11.2 points. At n = 60 the Wilson interval is the better choice, and it gives 17.1% to 39.0%; Chapter 16 derives why. The larger model's 2 of 60 has a Wilson interval of 0.9% to 11.4%. Both are small samples on one toy task, not a benchmark. They settle the qualitative point: enforcing a schema removed every format failure and none of the meaning failures. The smaller model read "mango" as another flavor and "7 tubs" as 1, in perfectly valid JSON.

That is why the validator below exists. A schema narrows what the model can say. Only a comparison against trusted facts says whether what it said is right.

The trusted side: an eval config the model never writes

Here is the application-owned config for this fixture. The model sees none of it as editable input.

Why 400 dev cases? At a pass rate near 50%, the widest case, the 95% half-width is 1.96·√(0.25/400) = 1.96 × 0.025 = 0.049. Four hundred cases buy an error bar of about ±5 points. Cut to 350 and it grows to 1.96·√(0.25/350) = ±5.2 points; the policy fixes the number so every run of the prompt is measured on the same cases.

The reference cost, computed from config alone:

• samples: 400 cases × 5 = 2,000;
• input: 2,000 × 1,200 = 2.4 million tokens × $3.00 = $7.20;
• output: 2,000 × 300 = 0.6 million tokens × $15.00 = $9.00;
• total: $16.20, or 1,620 cents, under the 2,000-cent budget.

No model call is involved. This is the answer the proposal will be checked against.

Five candidate plans

Every rejected row is valid JSON. Each fails because ordinary Python, running after the schema check, compared the plan with facts the schema never saw.

Where each check happens

View full-size diagram

Two properties matter. First, the config is loaded separately from the proposal. The proposal cannot edit the budget or the split rule, because those fields are not in its schema. Second, the comparison is plain code, not another model call. Anthropic's engineering notes on writing tools for agents (2025-09-11) observe that how a tool reports its failures shapes how well an agent can use it. A named error applies that idea here: case_count_mismatch and budget_exceeded call for different responses, and one generic "invalid" hides which. What to send back to the model for each code is its own design problem, worked through in a companion article on repair feedback.

Build the check

The function below assumes the proposal already passed type validation, for example with a strict Pydantic model. It runs as written on Python 3.12 with no dependencies.

def check_plan(runs, config):
    names = [run["eval_set"] for run in runs]
    if len(names) != len(set(names)):
        return "duplicate_run"
    registry = config["registry"]
    if not set(names) <= set(registry):
        return "unknown_eval_set"
    if any(run["split"] not in config["allowed_splits"] for run in runs):
        return "split_not_allowed"
    required = {name: registry[name]["dev"] for name in config["requested"]}
    proposed = {run["eval_set"]: run["n_cases"] for run in runs}
    if proposed != required:
        return "case_count_mismatch"
    samples = sum(n * config["samples_per_case"] for n in required.values())
    per_mtok = config["price_cents_per_mtok"]
    tokens = config["tokens_per_sample"]
    scaled = samples * (tokens["input"] * per_mtok["input"]
                        + tokens["output"] * per_mtok["output"])
    cost_cents = -(-scaled // 1_000_000)  # round up to a whole cent
    if cost_cents > config["budget_cents"]:
        return "budget_exceeded"
    return {"status": "accepted_draft", "cost_cents": cost_cents}


config = {
    "registry": {"reasoning-v3": {"dev": 400, "test": 1000},
                 "tool-use-v1": {"dev": 200, "test": 500}},
    "allowed_splits": {"dev"},
    "requested": ["reasoning-v3"],
    "samples_per_case": 5,
    "tokens_per_sample": {"input": 1200, "output": 300},
    "price_cents_per_mtok": {"input": 300, "output": 1500},
    "budget_cents": 2000,
}
good = [{"eval_set": "reasoning-v3", "split": "dev", "n_cases": 400}]
assert check_plan(good, config) == {"status": "accepted_draft", "cost_cents": 1620}
assert check_plan(good, dict(config, budget_cents=1620))["cost_cents"] == 1620
assert check_plan(good, dict(config, budget_cents=1619)) == "budget_exceeded"
assert check_plan([dict(good[0], split="test")], config) == "split_not_allowed"
assert check_plan([dict(good[0], n_cases=350)], config) == "case_count_mismatch"
assert check_plan([dict(good[0], eval_set="reasoning-v4")], config) == "unknown_eval_set"
print("Plan accepted at $16.20; one cent less of budget rejects it")

The arithmetic stays in integers. Prices are cents per million tokens, so the product is cents × 10⁶, and the ceiling division rounds up to a whole cent. Floating-point dollars would make the equality test below fragile.

Test the boundary, not only the happy path

A validator that has only seen good plans has not been tested at its job. Two cases need explicit tests.

The budget edge. The reference cost is 1,620 cents and the check uses >. A budget of 1,620 must accept and 1,619 must reject. Tests at $10 and $30 would pass under both > and >=, so they cannot tell you which one you wrote.

View full-size diagram

The empty request. If the task requests no eval sets, the correct plan is an empty list costing zero, and the validator should accept it. That case is easy to break by treating an empty plan as suspicious. It is equally easy to confuse "the task requested nothing" with "the registry failed to load". Those are different failures and must not share a code path; a registry read that fails and returns {} would make every eval set look unknown, or worse, make an empty plan look correct.

Check the decision

When the schema carries the policy

You could generate a schema from config that allows only "dev" and only 400. That moves the check into the schema; it does not remove the need for trusted state, because someone still has to show where 400 came from and which config version the schema was built from. Constrained decoding against such a schema also changes what the model says, not only whether it parses; Chapter 2 shows the grammar pushing a model's answer somewhere it would not have gone. Either placement can work. Pick the one whose data flow a reviewer can follow, and test duplicate runs, unknown sets, the split rule, exact counts and the budget edge wherever the check lives.

A passing local check establishes the result for the config you supplied. It does not show that the run launcher reads the same config version, or that the run stayed in budget. Keep the accepted draft, the launched run and its metered cost as three separate records.

What would prove this wrong

The claim here is that schema validity and correctness are different properties, and that the gap is large enough to matter. It would be wrong for your task if, with the schema enforced, valid-but-wrong answers were rare enough that their interval sits below the error you can tolerate. You can measure that directly:

1. Author 100 or more requests with expected answers written by someone other than the person writing the validator.
2. Run the model with the schema enforced, several samples per request, and score parse, schema and correctness separately. The book's experiment script is a template.
3. Report the valid-but-wrong rate with a Wilson interval, and treat repeated samples of one request as one case, not several; Chapter 16 explains why repeats are not new cases.

If the upper end of that interval is below what your downstream decision can absorb, the business-rule check is cheap insurance rather than a necessity. On the two small models the book measured, it was a necessity.

Why this matters in a lab

Applied-AI work routinely puts a model's structured output in front of code that acts on it: a tool call, a generated config, a grader's verdict. The skill is not writing a schema. It is naming, for each field, which facts it must agree with and where those facts live. An interviewer can probe this with one question: "Your structured output passes validation 100% of the time. Is the feature done?" A strong answer separates format from meaning, cites a measured valid-but-wrong rate with an interval, and says which trusted state the second check needs. A work sample shows the same thing in code: a reference computed without the model, named errors, and tests at the boundary.