A Valid JSON Object Can Still Hold the Wrong Number

OpenAI's Structured Outputs guarantees that the model's response adheres to the JSON Schema you supply. The same guide says the output may still contain mistakes. Parsing tells you the shape arrived. It does not tell you the number is right, or that it came from the paper at all.

Say you are extracting reported benchmark scores from 60 papers into a comparison table. Some papers don't report every benchmark. If the schema requires a number, the model has no valid way to say so, and a plausible number is the easiest thing to produce. The guide says all fields must be required, and shows how to emulate an optional one with a union type that includes null. Schemas also need "additionalProperties": false. So give absence its own representation:

{
  "type": "object",
  "properties": {
    "model": { "type": "string" },
    "benchmark": { "type": "string" },
    "split": { "type": "string" },
    "score": { "type": ["number", "null"] },
    "evidence": { "type": ["string", "null"] }
  },
  "required": ["model", "benchmark", "split", "score", "evidence"],
  "additionalProperties": false
}

Tell the model that evidence must quote the sentence or table cell the score came from, and that score and evidence are both null when the paper does not report the benchmark. Downstream, a score with no evidence is a finding to check, never a table entry, and null never turns into 0 or an empty string that later code reads as valid.

The check. Collect 20 papers, or write synthetic results tables, where you know the correct cell. Make at least 5 of them omit the benchmark you ask for. Then count per item:

• schema-valid, which should read 20 of 20, since that is what the feature promises;
• value-correct, the number you care about;
• fabrications (a number where the truth is null) and misses (a null where the paper reports one).

When a value is wrong, write down why. The usual suspects in results tables are the baseline's row instead of the proposed method's, the dev split instead of test, and a neighboring column. Those error types tell you what to change in the prompt or the schema, such as an explicit split enum.

Zero fabrications out of 5 is encouraging but weak. With zero events in n trials, the 95% upper bound on the rate is 1 − 0.05^(1/n): about 45% at n = 5 and 9.5% at n = 30, where it is already close to the rule-of-three shortcut, 3/n. Grow the not-reported set until that bound is one you would put your name to.

Two outcomes need their own branch in your code: the refusal field the guide shows for safety refusals, and a response whose status is incomplete because max_output_tokens ran out. Neither should be coerced into an empty record that looks like a result.

A table of prior results often feeds a paper, a comparison plot or a take-home write-up. Chapter 2 builds typed tool outputs and validation from scratch and separates what a schema guarantees from what it cannot. The course repo's chapter 2 exercises include a constrained-decoding notebook.

OpenAI documentation.