Evaluating Coding Agents: Acceptance the Agent Cannot Redefine
Decisions about coding agents rest on one number: how often the agent's change passes a check it could not influence, on tasks fixed in advance, with a 95 percent binomial interval. Lesson 1 derives it from the dark-factory idea, where the evaluation is the only review, and covers specification gaming and reward hacking. In lesson 2 you run a small Python harness with an acceptance suite pinned by hash. It shows four real OpenAI Codex runs, one with a defect its own eight passing tests missed and the protected suite rejected, a gamed candidate refused before it runs, and a rescoring after the suite was strengthened, and says what four runs cannot show.
OutcomeYou leave with: a small evaluation harness in plain Python that grades a real coding agent's work against a suite the agent cannot edit, rejects a defect the agent's own tests missed, refuses a candidate that games the check, and a plan to grow it into a task set with a reported interval
Rod Rivera
Professor
Desk
Every lesson is free. Lessons marked open need no account; the rest open once you register.
What You'll Learn
- Why an evaluation of a coding agent needs acceptance the agent cannot redefine, a fixed task set, and a pass rate with an error bar
- How to derive the 95 percent binomial interval for a pass rate, and why it fails at small n and near 100 percent
- Why an agent that can edit its own check will pass it, and how specification gaming and reward hacking name that failure
- How to keep an acceptance suite out of the agent's reach, and pin it so nobody edits it in passing
- Why the agent's own passing tests are recorded and never trusted
- Why a handful of runs is not a pass rate, and why every candidate is rescored when the grader changes
- What "dark software factory" means, and why it makes the evaluation the only review
Prerequisites
- Python, enough to read a 100-line script and a unit test
- A terminal and Python 3.9 or newer. No container engine, no API key and no account are needed for the opening unit
- Enough probability to follow a mean and a variance; the lesson derives the rest
- Helpful, not required. The Codex course in this track, or the Claude Code course, for operating a coding agent
Read the rest of Evaluating Coding Agents: Acceptance the Agent Cannot Redefine
The first lessons need no account. Sign in free to read the rest and keep your place on any device.
