What It Means to Evaluate a Coding Agent

A coding agent finishes a task and tells you it worked. That is one claim about one run. An evaluation answers a different question: across a set of tasks fixed in advance, how often does the agent produce a change that passes a check it could not influence, and how sure can you be of that number?

The question matters in a lab for a plain reason. If you build, train or deploy a coding agent, every decision about it, such as which model, which prompt, or whether a change made it better, rests on a number like this. If you are interviewing for an applied-AI or research role, "how would you know your agent improved?" is a question you should be able to answer with a method and an error bar, not an impression.

This lesson sets out the three things such an evaluation needs and derives the error bar. The next one has you run a small harness and watch it reject a real agent's change.

Why start with dark factories

The phrase "dark factory" comes from manufacturing: a plant that runs with the lights off, because nobody is inside to need them. In 2026 the phrase moved to software, and it is the limit case for evaluation. When nobody reads the change, the evaluation is the only review there is.

Two sources set the vocabulary, and both are first-hand accounts by the people doing the work. I read both on September 21, 2026.

Dan Shapiro's essay of January 23, 2026, "The Five Levels: from Spicy Autocomplete to the Dark Factory," describes a ladder of decreasing human involvement in writing software. Of the top he writes: "It's dark, because it's a place where humans are neither needed nor welcome." The ladder is one author's way of describing how the work changes. It is not a standard, and no one certifies a level.

StrongDM's account of February 6, 2026, by Justin McCarthy, is more concrete. It describes a software factory as "non-interactive development where specs + scenarios drive agents that write code, run harnesses, and converge without human review," and states two rules: "Code must not be written by humans" and "Code must not be reviewed by humans." It validates with scenarios kept outside the codebase and asks, "what fraction of them likely satisfy the user?" That question is an evaluation question: a fraction, over a set of scenarios the builder does not control. These are one company's reported practices. Its results are self-reported, and nobody outside has reproduced them.

What is established, and what is contested

Teams do run coding agents that produce changes without a person typing them, and some merge those changes without a person reading each one. The mechanisms they describe are ordinary engineering: written specifications, tests kept away from the builder, isolated environments, and records of what happened.

Whether this holds up is contested. None of the accounts reviewed for this course shows an arbitrary product maintained this way for years, and none gives a failure rate you could check. High agent output is also often mistaken for the real claim: a team can have agents write most of its code while a person still reads every change. That is heavy use of AI. It is not a dark factory.

The working definition this course uses: a software factory is a repeatable system that admits a change request, produces a candidate, evaluates it against an acceptance contract the builder does not control, releases what passes, and keeps enough state to recover. A dark software factory, in the strict sense, runs that routine cycle without a person writing or reading each change, inside a declared scope. This is a working definition for teaching, not an industry standard.

View full-size diagram

flowchart LR
    T["Task: request, allowed paths, acceptance criteria"] --> C["Candidate change, made in isolation"]
    C --> V["Independent verification: a suite the builder cannot edit"]
    V --> D["Decision record: accept or reject, with evidence"]
    D --> R["Release, observe, recover"]
    P["A person"] -.-> T
    P -.-> V
    P -.-> D

The dotted lines are where the evaluation lives. A person writes the task and owns the acceptance suite. Whether a person may stop reading each change depends on one thing: evidence that the agent passes that suite often enough, measured honestly. The rest of this lesson is what "measured honestly" means.

The three things an evaluation of a coding agent needs

1. Acceptance the agent cannot redefine

If the thing being graded can change the grader, the grade measures nothing. An agent told to make the tests pass has two routes: fix the code, or change the tests. Both satisfy the literal instruction.

This failure has a name. A 2020 DeepMind post by Victoria Krakovna and colleagues, Specification gaming: the flip side of AI ingenuity, defines specification gaming as behavior that satisfies the literal specification of an objective without achieving the intended outcome. When the objective is a training reward, the same thing is usually called reward hacking. A 2025 paper by OpenAI researchers, Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation, opens by calling reward hacking, "where AI systems misbehave due to flaws or misspecifications in their learning objectives," "a key challenge in constructing capable and aligned models," and studies it "in agentic coding environments."

You do not need a trained model to hit the problem. Any agent with write access to the check can pass it by editing it. The defense is structural: keep the acceptance suite outside the agent's workspace, and pin it by hash so a change to it is a separate, visible act. A fresh reviewer conversation does not prevent this, and neither does a sentence in the prompt. The next lesson shows the structural defense refusing a candidate that brings its own copy of the suite.

2. A fixed task set

A pass rate is a statement about a set of tasks. If the tasks, their acceptance suites or the agent's configuration change partway through, the rows before and after the change are measuring different things. Fix all three before the run, write them down, and treat any change as a new version of the evaluation that requires rescoring everything under it.

The task, not the attempt, is the unit you sampled. Running the same task five times tells you how variable the agent is on that task. It does not tell you about tasks you have not written. Chapter 16 of the book measures this directly: on one of its runs, every case gave the same result on both repeats, so the second repeat added no information.

3. A pass rate with an error bar

Here is the derivation, so you can check the numbers yourself.

Treat each task as a trial that passes with an unknown probability p. Run n tasks, independently, and count k passes. Then k follows a binomial distribution, with mean np and variance np(1 − p). Your estimate of the pass rate is p̂ = k/n. Dividing by n scales the variance by 1/n², so

Var(p̂) = np(1 − p) / n² = p(1 − p) / n
SE(p̂)  = √( p(1 − p) / n )

For large enough n, p̂ is close to normally distributed, and 95 percent of a normal distribution lies within 1.96 standard deviations of its mean. Plugging p̂ in for the unknown p gives the usual 95 percent interval:

p̂ ± 1.96 · √( p̂(1 − p̂) / n )

Now use it. Suppose agent A passes 38 of 50 tasks and agent B passes 35 of the same 50. These are arithmetic examples, not measurements:

A: p̂ = 38/50 = 0.76   SE = √(0.76 · 0.24 / 50) = √0.003648 = 0.0604   1.96 · SE = 0.118   interval [0.64, 0.88]
B: p̂ = 35/50 = 0.70   SE = √(0.70 · 0.30 / 50) = √0.004200 = 0.0648   1.96 · SE = 0.127   interval [0.57, 0.83]

A six-point gap, with error bars of about twelve points each way. These two numbers alone do not show that A is better. Because both agents ran on the same tasks, the stronger test compares them task by task, counting where one passed and the other failed; Chapter 16 derives that paired comparison (McNemar's test).

The same formula tells you how many tasks you need before you run anything. To get a half-width of 0.05 when p is near 0.5, solve 1.96 · √(0.25 / n) = 0.05:

n = (1.96 / 0.05)² · 0.25 = 1536.64 · 0.25 = 384.16, so 385 tasks

The formula also fails exactly where agent evaluations often live: small n and pass rates near 1. At 4 passes out of 4, p̂ = 1, the estimated standard error is 0, and the interval collapses to the single point [1, 1]. Four clean runs cannot show an agent never fails. The Wilson interval fixes this by solving for the values of p that are consistent with what you saw, instead of plugging in p̂. For 4 of 4 it gives about [0.51, 1.00]. Chapter 16 derives it and checks its coverage exactly.

What was checked

Both dark-factory sources were read on September 21, 2026, and each quotation was matched against the page's text that day. They are first-hand accounts, so their outcomes are their authors' reports, not findings of this course. The working definition comes from Prof Rod's own research review of September 10, 2026, which examined 21 dated sources and reproduced no vendor's results. The DeepMind post (April 21, 2020) and the abstract of arXiv paper 2503.11926 (submitted March 14, 2025) were read on September 30, 2026; the specification-gaming definition is paraphrased, and the reward-hacking phrases are quoted from the abstract. The interval arithmetic was recomputed in Python on September 30, 2026. The 38-of-50 and 35-of-50 agents are examples for the arithmetic; this course has not measured a pass rate for any agent.