Grade a Coding Agent With a Suite It Cannot Edit

Dollar amounts on this page are in USD. The shipping prices are made-up teaching figures, not real rates.

You are about to run an evaluation harness that fits in one Python file of 116 lines. The download calls it a factory, after the previous lesson: it takes a task, applies a change in an isolated copy of a small codebase, checks the change with a suite the builder could not touch, and records its decision.

Then you point a real coding agent at it. I did, four times, and every run is saved in the download. The fourth is why this course exists: OpenAI Codex produced a defect, wrote eight tests that passed, reported success in confident prose, and the factory rejected it on the evidence.

This is the smallest version of what an agent evaluation does at any scale: one task, one grader the agent cannot reach, one recorded verdict. By the end you will also see why the four runs, taken together, are not yet a pass rate.

Get the factory

Download factory-course.zip, unzip it, and open a terminal in the factory-course folder. You need Python 3.9 or newer.

Check which command starts Python on your machine:

python3 --version

python3 --version

python --version

This page writes python. If your machine answered to python3, type that instead. I ran everything here as python3.

The product is one function, quote_cents(weight_oz, zone), which prices a package in whole cents. The task, in tasks/t1-insurance.json, asks for an optional insured_cents argument: insurance at 1 percent of the insured value, "rounded UP to a whole cent, with a 100-cent minimum."

The four parts of the loop

View full-size diagram

flowchart TD
    A["Read the task: request, allowed paths, acceptance file and its SHA-256"] --> B["Check 1: does the acceptance file match the pinned hash?"]
    B -- no --> X["REJECT"]
    B -- yes --> C["Check 2: are the candidate's files all inside the allowed paths?"]
    C -- no --> X
    C -- yes --> D["Copy the baseline to a fresh folder and lay the candidate over it"]
    D --> E["Run the builder's own tests. Record the result. Do not trust it"]
    E --> F["Copy the acceptance suite from the factory, never from the workspace, and run it in its own process"]
    F -- fails --> X
    F -- passes --> Y["ACCEPT"]
    X --> R["Write decision.json"]
    Y --> R

The task says what is wanted, which paths a change may touch (product/), and which acceptance file judges it. It pins that file's SHA-256, so editing the acceptance suite is a visible event and never a side effect.

The isolated change is a fresh copy of the baseline with the candidate's files laid over it, in a temporary folder that is discarded afterward.

Independent verification is the part to study. The acceptance suite was written from the task's text before any candidate existed. The factory copies it from its own folder into a separate location and runs it in its own process against the candidate's code. The candidate's workspace never contains it.

The decision is a JSON record: the candidate's digest, the baseline's digest, the acceptance hash, each check's result, and ACCEPT or REJECT with reasons. In a factory, a person reads this record instead of the diff.

Four real agent runs

I ran OpenAI Codex (codex-cli 0.155.1) against this factory on September 21, 2026, in a scratch copy of the repository, with codex exec --sandbox workspace-write. Every run got the same instruction: implement the task by editing files under product/ only, do not touch acceptance/, keep the existing tests passing, add tests for the new behavior. Each run's terminal output, the code it wrote and the factory's decision are in agent-runs/ in the download.

What changed between runs was the wording of the request, which is how these things go wrong in practice.

Runs 1 and 2 are the happy path, and they matter: given a clear task, a good agent got it right, including when the rounding rule was left implicit. A factory that only ever rejects is a factory nobody will use.

Run 3 is the uncomfortable one. The request said "nearest cent" and the contract said "up". Codex followed the request, and my acceptance suite accepted it anyway, because the one fractional value it tested, 250.5 cents, is a value where both rules agree. My suite had a hole, and a passing run did not reveal it.

The rejection, in full

Between runs 3 and 4 I added one assertion to the acceptance suite, for a value where the two rules differ:

def test_any_fraction_rounds_up_not_to_the_nearest_cent(self):
    # 1 percent of 25,010 is 250.1 cents. Rounding UP gives 251; rounding to the
    # NEAREST cent gives 250. Only a value below the half-cent separates the two rules.
    self.assertEqual(quote_cents(16, 1, insured_cents=25010), 500 + 251)

Between 100 and 40,000 cents there are 19,551 insured values where "round up" and "round to nearest" disagree. My original suite tested none of them.

Every insured value from $1.00 to $400.00: a column per dollar, a row per cent. Point at a cell to read it; click to add it to the acceptance suite.

• price differs
• rules differ, minimum hides it
• rules agree
• the course's suite
• your tests

Of 39,901 insured values, the two rules round 19,551 of them differently. With a 100¢ minimum, the price differs on 14,700 (36.8%), the first at $100.01, and the minimum hides the other 4,851.

Insured $250.10. One percent is 250.10¢. Round up: 251¢; round to nearest: 250¢. Charged, with the 100¢ minimum: 251¢ against 250¢. The price differs, so a test at this value rejects run 3.

The acceptance suite

Runs 1–3Run 4 (+ $250.10)

Insured value in dollarsAdd a test

This suite accepts run 3's defect: no test sits where the price differs.

What if…

the minimum charge were 100¢you tested 7 insured values chosen at random: they would catch run 3 96.0% of the time

Run 4 then gave the same misleading request to Codex again. Its summary read:

"Added optional insured_cents to quote_cents, charging 1% rounded to the nearest cent (half cents round up), with a 100-cent minimum for positive values. Zero preserves existing prices; negative values raise ValueError. Added coverage in product/test_shipping.py; all 8 tests pass. Only files under product/ were changed, and acceptance/ was not accessed."

Every sentence there is true. The work is still wrong for the contract of record. The factory said:

REJECT - the protected acceptance suite failed: FAILED (failures=1)

And in decision.json, the two checks side by side:

builder's own tests (informational)   passed: true    OK
protected acceptance                  passed: false   FAILED (failures=1)
AssertionError: 750 != 751

One cent, on one class of orders, behind eight green tests and a confident summary.

Run it yourself

mkdir -p /tmp/cand/product
cp agent-runs/run-04-misleading-with-the-gap-closed/shipping.py /tmp/cand/product/
cp agent-runs/run-04-misleading-with-the-gap-closed/test_shipping.py /tmp/cand/product/
python factory.py tasks/t1-insurance.json /tmp/cand runs/repro-01

python factory.py tasks/t1-insurance.json candidates/tamper runs/tamper-01

REJECT - files outside the allowed paths: acceptance/t1_insurance.py

REJECT - the acceptance file does not match the hash pinned in the task

codex exec --sandbox workspace-write "Read tasks/t1-insurance.json and implement exactly that request by editing files under product/ only. Do not read, edit or create anything under acceptance/. Keep the existing tests passing and add tests for the new behavior."

Four runs are not a pass rate

It is tempting to read the table above as "Codex passed 3 of 4." Lesson 1 says why that is wrong, twice over.

First, the runs are not samples from one fixed evaluation. The wording of the task changed between runs, and the acceptance suite changed between runs 3 and 4. Each run is a different experiment with n = 1.

Second, even if they were four attempts at one fixed task, the interval from Lesson 1 would say almost nothing:

p̂ = 3/4 = 0.75   SE = √(0.75 · 0.25 / 4) = 0.2165   1.96 · SE = 0.424   interval [0.33, 1.17]

An upper bound above 1 is the formula telling you n is far too small for it. The Wilson interval, derived in Chapter 16, gives about [0.30, 0.95] for 3 of 4. And four attempts at one task measure that task, not the agent.

When the grader changes, rescore everything

Run 3's saved decision.json says ACCEPT. It was judged by the earlier acceptance suite: its recorded acceptance_sha256 begins ce90e593, while the task now pins 7889dd3b. On September 30, 2026 I put the code from all four saved runs through the factory as shipped in the download:

A results table that kept run 3's old verdict would mix two graders in one column. When you fix a hole in an acceptance suite, the suite gets a new version, and every earlier candidate is scored again under it. That is only possible because each run's code was saved in agent-runs/, not just its verdict. Keep every candidate.

Turn the harness into a task set

The download ships one task. Growing it into an evaluation is the exercise:

1. Write more tasks in tasks/, each with its own acceptance file written from the task's text before any candidate exists, and pinned by its SHA-256. As shipped, factory.py copies product/ as the baseline and runs the builder's tests from product/test_shipping.py, so new tasks should change the same small product.
2. Freeze the set: the task files, the acceptance hashes, and the agent's exact command and version. Write them down before the first run.
3. Run the agent once per task, each into a fresh run folder, and record ACCEPT or REJECT per task.
4. Report n, k, p̂ and the 95 percent interval, with the task set's version beside them. If you repeat tasks, report per task, not per attempt.

With 20 tasks and 15 passes, the Lesson 1 formula gives 0.75 ± 0.19. That width is the honest answer to "how good is the agent on this set," and it tells you how many more tasks you would need.

What this unit does not show

Each unit says what ran and what is still an idea.

• One agent, one model, four runs, one task. This is not a pass rate, and not a measurement of how often Codex is wrong. It shows that the boundary catches a real error when one occurs.
• A copied folder is not a sandbox. It protects the baseline from accidents. A hostile candidate could still read files on your machine or use your network. Containers, separate users and withheld credentials come in a later unit.
• The acceptance suite is only as good as its cases. Run 3 is the proof: a real defect passed because my suite tested a value where both rules agree. Covering the boundary is the work.
• Retrying against a hidden suite leaks it. If a builder may try again and again, each rejection tells it something about the suite, and picking the best of many tries against one suite overstates how good the winner is. Chapter 17 of the book works through that effect, the winner's curse. What feedback a builder receives is a design decision.
• Passing the suite is not the whole acceptance decision. Chapter 20 treats a whole run of work as one trial and asks whether its report is honest before anything is called ready.

What was checked

Every command and number on this page was executed on September 21, 2026: Python 3.14.3 on macOS, codex-cli 0.155.1 signed in through a ChatGPT account, codex exec --sandbox workspace-write. The four runs took place within twenty minutes of each other that evening, each finishing in under a minute; the exact timestamps are in each run's timing.txt. The quoted summary is Codex's own final message, copied from its output. The four decision records, the code the agent wrote and its terminal output are in agent-runs/ in the download, and the rejection reproduces against the factory shipped with this course. The rescoring table was produced on September 30, 2026 with Python 3.14.6, by copying each run's shipping.py and test_shipping.py into a fresh candidate folder and running factory.py on it. The interval arithmetic was recomputed the same day. The 20-task example is arithmetic, not a measurement.