The test suite that proves nothing
An agent refactors your evaluation script. Every test passes. In the same change, it regenerated the tests' expected values by running the new code. The suite is green and it has checked nothing about whether the refactor kept your results.
This is the check to learn first when you use a coding agent such as Claude Code on research code or a take-home: before you ask whether the tests pass, ask where their expected values came from. The rest of this guide builds around it. It applies to a paper reproduction, an eval harness, a notebook becoming a package or a change to an agent you are studying.
The skill is in demand. On October 2, 2026, the jobs board found working with coding agents mentioned in 28% of applied AI postings and 6% of research engineer postings, and research postings ask for it in other words: Anthropic's Code RL posting, read on September 30, 2026, wants people who can "build the reward signals and verifiers that capture what 'good code' means."
The agent has hands
Autocomplete suggested, and nothing happened to your files until you accepted. An agent with tool access reads files, runs commands and edits within its configured authority. Claude Code's permission modes and rules set that boundary.
A permitted action changes the working state before you review the result. A permission prompt tells you an action was allowed. It does not tell you the result is correct.
Two consequences for research work. A short request such as add a calibration metric to the eval harness may touch the loader, the scorer and the report, so state which inputs, which output, and which existing numbers must not change. And a second attempt is cheap to request but not free: what an agent loop costs derives the bill, and both candidates still need checking.
Run the regenerated-tests check
A test compares output with an expectation. If the expectation came from the code under test, the comparison shows only that the code is deterministic. Three steps make the check routine.
Before reading results, list the test and fixture files the change modified against the commit you started from:
git diff --name-only <base-commit> -- tests/ fixtures/Any expected-value file on that list needs a reason you can state.
Check out the base commit in a separate worktree, run both versions on the same frozen inputs, and compare their outputs row by row. Differences are either the intended change or a regression, and you should be able to say which for each one.
Expected values fixed before the change, or computed by hand from a small case, are evidence. Values the new code produced are a snapshot. Label which is which in the test file.
Three constructed examples, not incidents from a real project: an eval script that reports accuracy on the validation split instead of the test split; a deduplication step that silently drops 40 rows of the eval set; and a refactor whose tests pass because the expected values were regenerated from the new code.
Each needs a different check: the split named and asserted in code, a count reconciliation from input to output, and expected values fixed independently of the implementation. A clean table is not evidence that any of these passed.
Check the decision
Who can change a coding agent's tests takes the same question to the level of permissions. A green suite still leaves the interface to inspect: run the changed page and check what a reader can actually see and do.
Four habits that make delegation checkable
These habits reduce particular risks; they do not make every task safe to delegate.
Choose a task whose acceptance cases you can establish independently. For deduplicating an eval set, define which items count as duplicates, which near-duplicates must stay, and how every input item is accounted for. A row count alone cannot show the right items survived.
Work on a branch and keep the original. Claude Code's checkpoints can rewind edits made with its file tools, but the docs say they do not track files changed by shell commands and are not a replacement for version control. Neither undoes a job submitted to a cluster or a result posted to a shared tracker.
A summary can omit a changed test or an unrelated edit. The diff shows what changed; it does not show that the code computes the right number. Run the acceptance cases and look at the output.
Separate a data fix, a restructuring and a renaming when each can be reviewed alone, so an unexpected difference in a result can be traced.
Some work needs stronger boundaries. A number that goes into a paper or a decision about a training run, and that nobody recomputes. A dataset version that three experiments already consumed. A choice of baseline or ablation that lives only in your head. Keep acceptance with someone who can judge it, price the recovery path before granting write access, and write the unwritten constraint into the brief: if the eval must use the test split frozen before a model was trained, the agent has no way to know that unless you say so.
In a take-home or work sample
If a take-home allows a coding agent, the work product is the same as without one: code that computes the right thing, and evidence that it does. Say in the README which parts the agent wrote and which checks you ran. Expect questions like these:
- How do you know the metric uses the test split?
- Where did each expected value in your tests come from?
- What did you check that the agent's summary did not tell you?
- Which change in the diff would you revert, and why?
A candidate who answers from tests, diffs and reconciled counts shows the judgment that research engineer and applied AI engineer postings describe.
Did it save time? Measure, do not guess
Faster generation can move the slow part into review. Compare one contained task with your usual workflow under the same acceptance standard. Record specification, implementation, review, rework and handover time, plus token charges, and keep the model and tool versions with the result. Repeating a task you just did gives the second run a learning advantage, so a single before-and-after run is a probe, not a causal study.
Published evidence shows how far impressions and measurements can diverge. In METR's early-2025 randomized trial, experienced open-source developers took 19% longer on tasks where AI tools were allowed, while expecting a speedup. When to let an agent run unsupervised covers that study, its follow-up and their limits.
This site has not run this probe yet. Until it does, treat any time saving here as a hypothesis to test on your own work.
The chapter runs a full day of the book's agent against acceptance checks and shows how to report what passed, what failed and what stayed unknown.
