Gate a Codex Step on Its Status, Its Schema and Its Numbers

A pipeline of five steps that each pass a bad artifact along 5% of the time delivers a clean result only 0.95⁵ ≈ 0.77 of the time. Nearly one night in four, the morning summary would rest on something broken. Chapter 3 derives the pⁿ behind that. A gate between steps stops the compounding, and for a Codex step it has to answer three questions: did the run succeed, is the report the right shape, and are its numbers true.

Take a nightly job that compares last night's eval scores with the baseline and writes a triage report for the team's regression dashboard. OpenAI's non-interactive guide documents the pieces. codex exec sends progress to stderr and only the final agent message to stdout. -o writes that message to a file. --output-schema requests a final response that conforms to a JSON Schema you supply. With --json, stdout becomes a JSON Lines stream of events, including turn.completed and turn.failed. The default sandbox is read-only, which is all a report needs.

rev=$(git rev-parse --short HEAD)
codex exec --json --output-schema triage.schema.json -o "reports/triage-$rev.json" \
  "Compare runs/latest with runs/baseline for commit $rev. Report every metric that moved." \
  > "logs/triage-$rev.jsonl"
status=$?
if [ "$status" -ne 0 ] || grep -q 'turn.failed' "logs/triage-$rev.jsonl"; then
  echo "codex step failed; not publishing" >&2; exit 1
fi
uv run python scripts/check_triage.py "reports/triage-$rev.json" runs/ "$rev" || exit 1

The schema fixes the shape. Each finding needs task, metric, before, after, delta, run_id and a disposition from a fixed list (regression, noise, improvement, needs_rerun), and the report carries its revision. A heading that drifts from "Regressions" to "Things to watch" can no longer break the parser, and the file name tells whoever reruns the suite which code it describes.

The schema cannot check the content, so check_triage.py does. It recomputes after − before and compares it with delta, confirms every run_id exists under runs/, checks that the revision inside matches the file name, and exits nonzero on any mismatch. Chapter 2 derives what a schema does and does not guarantee, from constrained decoding up.

If the job reads runs from an experiment tracker through an MCP server, mark that server required = true. The guide says an enabled required server that fails to initialize makes codex exec exit with an error instead of continuing without it. That turns a missing tool into a status the gate already reads. A server that starts may still lack access, for example with an expired token, so make the job's first step one harmless read of a run id you know exists. Chapter 14 shows why the client, not the protocol, decides what runs.

Then break the gate on purpose. Each of these should stop the dashboard step:

• a report whose delta is wrong (if it passes, the checker is reading the model's arithmetic instead of redoing it);
• a report that cites a run id that does not exist;
• a scratch config whose tracker server points at a binary that does not exist;
• a run you interrupt with Ctrl-C partway through. This note does not claim what Codex leaves behind on interruption; the test is how you find out.

Keep one correct report that must pass, so a gate that rejects everything does not count as working.

A passing gate says the report is well formed and agrees with the logs it cites. It does not say a regression is real; that takes the paired comparison and error bars in chapter 16. Still, a well-formed report with an invented run id is worse than a crash, because someone will act on it.

OpenAI documentation.