Give an Unattended Codex Run a Finish Line You Can Check
"Keep improving the eval harness" has no finish line, so an unattended run can wander until something outside it stops it. Give the job one bounded question with two allowed endings.
Take a nightly check on a research repo: does results/table.csv have one row per model in the sweep config? The run ends accepted with its evidence, or blocked with the command that shows the problem. Write those two endings into a schema:
{
"type": "object",
"properties": {
"status": { "type": "string", "enum": ["accepted", "blocked"] },
"evidence": { "type": "string" },
"repro_command": { "type": ["string", "null"] }
},
"required": ["status", "evidence", "repro_command"],
"additionalProperties": false
}codex exec --output-schema ./run-status.schema.json -o ./run-status.json \
"Check that results/table.csv has one row per model in configs/sweep.yaml. Do not edit files."OpenAI documents that codex exec prints progress to stderr and only the final message to stdout, that --output-schema requests a final response conforming to the schema, and that -o writes that response to a file. The default sandbox for codex exec is read-only. The documentation does not describe a time or turn limit for codex exec, so enforce the wall-clock limit in the scheduler that launches it.
The check. Build two fixtures: a table that matches the config and one with a model missing. Run each 10 times and count how often status matches the fixture. Ten out of ten is encouraging, not proof. With zero misses in n runs, the 95% upper bound on the miss rate is 1 − 0.05^(1/n): about 26% at n = 10 and 6% at n = 50. For large n that is close to 3/n, the rule of three. Pick n from the miss rate you can live with.
Repetition is what separates this check from the gate on a Codex step's output. That gate is code: it behaves the same on every run, so one bad fixture tests it. The status here is the model's judgment, which can change from run to run, so one run per fixture tells you almost nothing about its miss rate.
A loop that stops for a reason you can read is the subject of Chapter 3 of the book.
