When to Let a Coding Agent Run Unsupervised, Using Public Evidence

If you build or evaluate agents, you will be asked a version of this question: when is it reasonable to let a coding agent work without watching it? The answer is a short operating contract for one task: what it may change, what proves it worked, how to undo it, what it may spend, and who reviews it. This article grounds that contract in public evidence and in the mechanisms Claude Code actually documents.

What public evidence says about autonomy

Three published results from METR are worth knowing well, including their limits.

Task length predicts success. METR's time-horizon work (paper) measures the length of tasks, timed on human experts, that a model completes with a given probability. The 50% time horizon is the human task length at which the model succeeds half the time. METR reports that this horizon has doubled roughly every seven months over six years, and that the models it studied succeeded almost always on tasks taking humans under four minutes and less than 10% of the time on tasks over about four hours. The authors discuss external validity and measurement limits; the absolute values depend on their task suites.

People misjudge their own speedup. In METR's early-2025 randomized trial (paper), 16 experienced developers worked on 246 issues in large open-source repositories they knew, with AI tools allowed on a random subset of issues. With AI allowed they took 19% longer. They had expected a 24% speedup, and afterwards estimated they had been sped up by 20%. The authors state that the result does not show AI fails to help most developers or that it holds outside their setting.

Measurement itself gets harder. METR's February 2026 update describes a later study with 57 developers. Its estimates were −18% for returning developers (confidence interval −38% to +9%) and −4% for newly recruited ones (−15% to +9%), on the scale where the original result was +19%, so negative means less time. METR calls the data "only very weak evidence" for the size of the effect, because 30% to 50% of developers said they chose not to submit some tasks they did not want to do without AI, which removes exactly the tasks where AI might help most.

Why long runs fail: reliability compounds

A long task is many decisions in a row. If each step succeeds independently with probability p, a run of n steps succeeds with probability pⁿ. Chapter 3 of the book derives this and measures where a real loop fails.

Two things change the arithmetic. Recovery: a step that fails and is detected can be retried, so what compounds is the rate of undetected failure. And checks: a test that catches a class of mistakes turns a silent failure into a retry. Both depend on what can check the outcome, which is the first axis of the decision.

Two questions to start the decision

View full-size diagram

Start with two properties of this task:

Can a machine check the outcome that matters? A test suite, a type check, a schema check or a comparison with frozen expected values each establishes one property. Name it and its blind spots. A passing unit test says nothing about whether the eval uses the right split.

How cheaply can it be undone? A branch, a worktree, a committed baseline. Not "will it go wrong" but "what does wrong cost." Submitting a job to shared compute or writing to a shared dataset is not undone by a branch.

The grid is a working heuristic, not a validated predictor. Record why a task sits in its cell and what evidence would move it.

Six operating modes

Higher numbers move supervision further from the individual action. They add coordination obligations; they do not show greater competence.

Change modes after representative checks support it. Familiarity is not enough.

Budget review before adding workers

Generation can outpace review. Measure the time to inspect results, resolve disagreements and integrate accepted changes. Claude Code's agent teams are experimental and disabled by default; the costs documentation says they use about seven times the tokens of a standard session when teammates run in plan mode, and the teams page suggests starting with three to five teammates and notes that three focused teammates often outperform five scattered ones. It also warns that two teammates editing the same file leads to overwrites.

If you cannot say where the review capacity for a fleet comes from, you have designed a backlog.

Decomposition that survives the usual failures (stopping at "looks done," drifting out of scope, duplicating a sibling's work, degrading as context fills):

• One task per dispatch, not one theme.
• Disjoint file ownership per worker.
• A brief that carries the objective, output format, tool guidance and boundaries.
• A check that rides with the task, in the prompt or as a gate outside it.
• A scope fence in the reviewer's brief: nothing outside the task changed.

Failure modes to test before handover

Check one task before delegating it

Check the decision

What an interviewer would ask

Expect to be asked how you would decide an agent's level of autonomy for a given task, and how you would know a coding agent helps at all. A strong answer cites evidence with its limits (task length predicts success, perceived and measured speedup diverge, selection can bias a comparison), then names the check, the undo path, the budget and the stop condition for the specific task. It treats a randomized comparison on a fixed task set as the way to settle the question, not an impression.

Register of claims to recheck

Sources reviewed 2026-09-30. Documentation describes contracts; it is not a receipt that every mode was run here.