The claim you will be asked to check
"Our team got 40% faster after adopting a coding agent." You will hear claims like this, and in an applied-AI or research role you may be asked to produce or audit one. The two ways it goes wrong are simple to state: the timer does not cover the whole task, and the tasks measured before and after are not the same kind. This article works both through with small constructed numbers you can check, then turns them into a measurement design.
Count the whole task
Take a routine task: write a failing test that reproduces a bug report. Suppose, as a constructed example, it takes 12 minutes by hand from reading the report to a reviewed test. With an agent, generation takes 3 minutes. That is the number that gets quoted. But in this serial workflow four phases sit on the path to an accepted test:
| Phase | What happens | Constructed minutes |
|---|---|---|
| Setup | Point the agent at the report, the module and the test conventions | 6 |
| Work | The agent drafts the test (the part usually reported alone) | 3 |
| Review | A person checks that the test fails for the right reason | 4 |
| Repair | Fix the draft and land it | 2 |
| Total with the agent | 15 |
Scroll horizontally to view columns.
Fifteen minutes against twelve: three minutes slower on this run. "Three minutes instead of twelve" compared a fragment of one process with the whole of the other.
Setup often amortizes. If the task recurs 40 times and setup happens once, the assisted total is 15 + 39 × 9 = 366 minutes against 40 × 12 = 480 by hand, a modeled saving of 114 minutes or 23.75%. That depends on all 40 tests being accepted with one setup and nine recurring minutes each, which is the thing to measure.
The agent may generate while a person does other work. Count that interval for time to completion, but measure attention separately before claiming labor saved. If phases overlap, use start and finish timestamps rather than summing durations.
Check the decision
Case mix can manufacture a saving
The second error changes which tasks are measured. Suppose review of agent pull requests takes 5 minutes for a routine fix and 20 for a hard one, in both months. In month A, 80 of 100 tasks are routine: 80 × 5 + 20 × 20 = 800 minutes, 8 per task. In month B, 95 are routine: 95 × 5 + 5 × 20 = 575 minutes, 5.75 per task.
The average fell 28%, and nothing about any task changed. Reweight month B to month A's 80/20 mix and you get 8 minutes again. The gain came entirely from fewer hard tasks entering the pool, perhaps because hard ones were routed to someone else.
Simpson's paradox: faster overall, slower everywhere
A mix shift can do more than exaggerate a gain. It can reverse the sign. Here is a constructed example, invented to show the arithmetic and not measured from any team. Fifty tasks are done without the agent and fifty with it. Tasks are easy or hard.
| Condition | Easy tasks | Hours each | Hard tasks | Hours each | Mean hours per task |
|---|---|---|---|---|---|
| Without agent | 20 | 1.0 | 30 | 4.0 | (20 + 120) / 50 = 2.80 |
| With agent | 45 | 1.1 | 5 | 4.4 | (49.5 + 22) / 50 = 1.43 |
Scroll horizontally to view columns.
The headline says the agent cut mean time from 2.80 to 1.43 hours, 49% faster. Within each kind of task the agent is 10% slower: 1.1 against 1.0 hours on easy tasks and 4.4 against 4.0 on hard ones. The aggregate reverses because the agent condition received mostly easy tasks.
This is Simpson's paradox: an association in every subgroup reverses when the subgroups are pooled, because a third variable, here task difficulty, is related to both the condition and the outcome. The fix is to compare at the same mix. Standardize the agent condition to the baseline mix of 20 easy and 30 hard tasks:
(20 × 1.1 + 30 × 4.4) / 50 = (22 + 132) / 50 = 3.08 hours
At the same mix the agent takes 3.08 hours against 2.80, 10% slower, which agrees with every subgroup.
If reweighting both conditions to the same task mix erases or reverses the gain, the gain came from which tasks were measured. Report per-stratum counts and times alongside any pooled number.
Check the reweighting
The real version of this problem
This is not only a classroom construction. METR's February 2026 update on its developer-productivity experiment reports that 30% to 50% of developers said they chose not to submit some tasks because they did not want to do them without AI. That is selection on exactly the variable that matters: the tasks where AI helps most leave the comparison. METR concludes its late-2025 data is "only very weak evidence" for the size of the speedup, and that the true speedup could be much higher among the tasks selected out. When to let an agent run unsupervised summarizes METR's earlier trial and its limits.
Design the measurement so mix cannot fool you
This is a protocol; it has not been run for this article.
Write what a correct output is for each task type, such as the test fails before the fix and passes after, with no test file modified. A quick, incomplete output does not count as done.
Assign the agent or no agent per task by coin flip after the task is chosen, as METR's trial did per issue. Randomization balances difficulty across conditions on average, including kinds of difficulty you did not think to record.
Label each task's stratum before work starts, from a rule you wrote in advance. Report counts and times per stratum, then a pooled estimate standardized to a declared mix.
Record tasks declined, abandoned or not submitted in each condition. If one condition loses more tasks, selection can decide the result, as it did in METR's follow-up.
Report the number of tasks, per-stratum means and an interval for the difference. Timing distributions are skewed, so compare ratios of times or medians, and use paired comparisons where the same task can be done both ways.
Chapter 16 of the book derives error bars and paired comparisons for exactly this kind of estimate.
What an interviewer would ask
Expect to be given a before-and-after table and asked whether the tool helped. A strong answer asks where the timer started and stopped, whether the acceptance rule stayed fixed, what the task mix was in each condition, and whether any tasks left the sample. It computes a standardized comparison on the spot. In a lab the same reasoning applies whenever an eval set changes composition between two model versions: a pooled score can rise while every category falls.
What would change the conclusion
A reported uplift is false for a comparable task if the whole assisted time exceeds the manual time. It is false at a stated volume if setup amortized over that volume still leaves the assisted total higher. And it is an artifact of case mix if reweighting both conditions to one mix erases or reverses it. Anyone with the per-task, per-phase and per-stratum records can run all three checks, which is why a baseline should keep them rather than a single average.
The chapter treats an evaluation as a measurement: binomial error bars, paired comparisons on the same cases, how many cases you need, and pass@k.
