Working with coding agents

Claude Code for AI Research

Twelve lessons on running Claude Code on research code so every number it reports arrives with a command, a commit, a seed, an n and an error bar, practiced on the book's public evaluation code.

OpenAI Codex for Research Engineers: First Task to Reviewed Change

The check to run before a coding agent touches research code: record a baseline, give the OpenAI Codex CLI one bounded change, read the diff, and compare the outputs yourself. You practice on a small stand-in repository, not an eval repo. Two lessons, both open without an account.

Agentic Coding with Cursor for Research Engineers

Review a Cursor agent's edit to research code the way a lab reviewer would: which split it reads, what the metric computes, and whether the run repeats.

Evaluating Coding Agents: Acceptance the Agent Cannot Redefine

Evaluate a coding agent the way a research team would need to: a fixed task set, an acceptance suite the agent cannot edit, and a pass rate with an error bar. Two lessons are available now: the method with the binomial interval derived, then a harness you run that rejects a real agent's defect.

Chapter 20: A whole day: reliability, honest reports and readiness

Budget a day's reliability step by step, bound what one passing day proves, and measure whether a model writes a faithful day report. Then run the whole shop day with injected failures.

Chapter 21: Context engineering: a budget for what the model sees

Decide what Lucy's model reads. Measured on small models and Claude: small models lost facts by 16,000 tokens, Claude found all to 150,000, and only pinned records held on every model.

Review a Month of Coding-Agent Work by Accepted Outcomes

Judge a month of agent work from every attempt, not only the accepted ones: fix the denominator, compare within task families, put error bars on each rate and keep unresolved jobs in the ledger.

A Coding Agent That Writes Its Own Tests Is Grading Itself

When a coding agent writes both the change and the tests, a passing suite measures the agent's idea of done. Treat it as an eval problem: tests written from the spec, exit codes, rendered measurements and a named human verdict.

Measure Whether a Coding Agent Weakens Tests Under Pressure

Coding agents sometimes pass by editing the tests instead of the code. Here is the evidence, the arithmetic, and a protocol you can run to measure how often it happens.

What You Still Decide When a Coding Agent Builds the Change

A coding agent can implement a change to your eval runner. It cannot decide what counts as the same request, what a retry may do, or who may change the tests.

Measure Coding-Agent Uplift Honestly: Case Mix and Simpson's Paradox

A coding agent can look faster overall while being slower on every kind of task. Count the whole task, hold the case mix fixed, and see how Simpson's paradox produces a false speedup.

Test What a Claude Code Permission Rule Actually Allows, in a Sandbox

A Claude Code permission rule is a string pattern, not an intent. Write the commands it should stop, run them through a headless session in a throwaway VM, and count the unintended allows before you trust the rule.

In a Dark Factory, Checking Is the Real Price of an Answer

As model calls get cheaper, what a dark software factory's output costs comes to be set by review time and acceptance rate, not tokens. The arithmetic, the error bar that decides a route, and the accounting that keeps it honest.

Read a Coding Benchmark Score: Contamination, Validity and Error Bars

A benchmark score is an estimate from a sample of tasks, graded by an oracle that can be wrong, on data a model may have seen. Put error bars on it and test contamination and oracle validity.

Clean an Eval Set With a Coding Agent, Then Prove It Dropped Nothing

An eval set merged from three sources has mixed label spellings, near-duplicate prompts and empty references. A coding agent will tidy it. Three counts, computed by your own script, show whether it dropped a row that would fake a gain.

What an Agent Loop Costs: Quadratic Resend, Cache Reads and Cost per Solved Task

Every call in an agent loop resends the whole context, so input grows with the square of the calls. Derive the cost with caching, price the solved task with an interval, with Claude Code as the worked instance.

Using a Coding Agent on Research Code: Ask Where the Expected Values Came From

An agent can refactor an eval script and pass every test because it rewrote the expected values. A short guide to using Claude Code or any coding agent on research code and take-homes, built around that check.

When to Let a Coding Agent Run Unsupervised, Using Public Evidence

Decide how much autonomy a coding agent gets from what the task can be checked against, how cheaply it undoes, and what METR's published studies do and do not show.

Check a Codex Fix Against the Original Failing Case

Your eval harness scores cases with an empty reference answer as correct, which inflates accuracy by the share of such cases.

Give an Unattended Codex Run a Finish Line You Can Check

"Keep improving the eval harness" has no finish line, so an unattended run can wander until something outside it stops it.

Keep API Keys Out of a Codex Skill Your Team Shares

A Codex skill that launches an eval run against a hosted model needs an API key. The skill itself should never contain one.

Count Every Row Before You Trust a Dataset Pull

You ask Claude Code to pull last month's eval runs from your tracker through an MCP server, or to page through an API for a dataset.

Test Your Results Scripts on Awkward Run Names

A results script that aggregates runs/.jsonl works on clean names. Sweeps don't always produce clean names: lr 3e-4.jsonl, a file that starts with a dash, a…

Run It Logged Out Before You Polish the Plots

A notebook can produce clean figures for you and fail on its first cell for anyone else. The difference is usually state you carry and a new reader does not…

Test the Date Cutoff in Your Contamination Filter

One common way to reduce contamination in a coding or math eval is a date filter: keep only problems published after the model's training cutoff.

Ask Claude to Run the Data Prep Twice Before You Trust the Row Count

After a crash, someone reruns the eval-set builder. If the script appends instead of replacing, the test set now holds every example twice, and nothing fails.

Ask Codex to Trace a Generated Results Table to Its Source

The results table in your paper repo shows 71.4 where the logs say 74.1. Editing the table by hand fixes the PDF until the next regeneration, and until then…

Turn a Repeated Results Check Into a Tested Codex Skill

Every time a sweep finishes, you paste the same instructions: check that each results file has the required fields, that no item id appears twice, and that…

Gate a Codex Step on Its Status, Its Schema and Its Numbers

A pipeline of five steps that each pass a bad artifact along 5% of the time delivers a clean result only 0.95⁵ ≈ 0.77 of the time.

Pilot a Codex Migration on Two Files, Then Diff the Requests

You need to move 40 experiment configs to a new format: a renamed field, a new default, a changed prompt-template path.

Give a Codex Skill One Request It Must Refuse: Publishing

A skill that builds a results table from eval logs looks excellent on the request it was written for.

Give Codex Write Access to the Repo Without Giving It the Network

Fixing a bug in an eval script needs write access to the repository. It does not need the network. But if the script's tests call a paid model API, or…

Pin the Working Directory of a Scheduled Codex Run

A reviewer asks where a number came from. The answer you want is a commit and a command. A nightly eval that runs Codex from cron or CI starts wherever the…

Parse Codex's Final Result, Not Its Progress Stream

A sweep driver calls codex exec once per config and builds a results table. If it parses everything the command prints, two things go wrong.

A Codex Worktree's Result Depends on Files Git Doesn't Carry

Codex ran your ablation in a separate worktree and reports a two-point gain. Before that number goes in a results table, to a teammate or into the trash with…

Compare Cost per Accepted Result, Not per Token or per Session

A cheaper model, or a cheaper Claude Code setup, can cost more per result you keep. A call or a session whose output you throw away cost money and bought…

Keep Your Claude Code Overrides Out of a Shared Research Repo

A research repo shared by four people needs the same permissions and hooks for all of them. It does not need your model override, the path to your scratch…

Prove a Claude Code Hook Blocks Before You Trust It

Say you write a PreToolUse hook so Claude Code can never edit data/test/, the held-out split your reported numbers depend on.

Check What Bash(uv run *) Actually Allows Before You Grant It

An allow rule is a threat model in one line. Before you save Bash(uv run ) so Claude Code can run experiments without prompting, write down the worst command…

Test the Environment Your Reviewer Will Actually Run

Your eval passes in the virtual environment you have been editing for three weeks. That environment holds packages you installed by hand, a cached dataset and…

Give Claude a Failing Eval Fixture Before You Ask for a Fix

"The accuracy looks too high" gives Claude Code nothing it can check. A three-row fixture does. Suppose the eval set was merged from two shards and some…

Test the Two-Correction Rule Instead of Taking It on Faith

The Claude Code best-practices guide gives a specific threshold: "If you've corrected Claude more than twice on the same issue in one session, the context is…

Prune Your CLAUDE.md, Then Test Whether Adherence Moves

Three hundred lines of instructions, and Claude still ignores the rule about where results files go. So you add the rule again in capitals.

Delegate Codebase Discovery, Then Spot-Check the Handoff

Finding where an unfamiliar research repo computes pass@k, or where it filters the eval set before scoring, can take dozens of file reads before you reach the…

Put the Check Your Numbers Depend On in a Hook

A line in CLAUDE.md such as "rerun the scorer tests after editing metrics.py" is followed at some rate.

A Stop Hook Gates the Turn, Up to a Cap

The session reports that the experiment code is done. The smoke test is red. A Stop hook turns that gap into something the session cannot talk past, within a…

The 7x Agent-Team Figure Applies to Plan Mode: Measure Your Own Ratio

The slug of this note says agent teams cost seven times more. The documented claim is narrower, and the difference matters when you plan compute or explain a…

Fan Out a Research-Repo Migration One Bounded Run at a Time

You need to migrate 200 experiment configs to a new schema, or move 200 eval tasks onto a new harness API.

Give Your Ablation Recipe Arguments, Then Check Each Run

You keep typing the same experiment recipe: train with this config and seed, evaluate on the fixed 200-case subset, write the results with the commit hash.

Pick a Subagent's Model by Measuring It on Your Own Tasks

The slug says run cheap subagents and reserve Opus for the hard part. That is a reasonable hypothesis, not a rule.

Measure What Survives Compaction, Then Write the Rest Down

Three hours into chasing a loss spike, you hold ten facts that matter: the seed that reproduces it, the step where it starts, the config hash, the…

Worktrees Separate Files, Not Experiments

You want two agents at once: one refactoring the data loader, one adding an ablation flag. Separate git worktrees give each its own directory and branch, so…

Measure What a Second Model Catches Before Adding It to Review

Measuring one reviewer means its recall on planted bugs and its precision on real findings. This note asks a different question: does a second reviewer catch…

Look at the Rendered Figure, Not Just the Exit Code

Claude Code edits the script that draws your results figure. It exits 0. The figure can still be wrong: error bars missing, a legend covering the curve you're…

Auto Mode Decides; the Sandbox Enforces. Test Which One Stops What

An agent running your eval harness reads untrusted text all day: benchmark prompts, scraped pages, other models' outputs.

Run /init, Then Test Every Command It Wrote Down

/init drafts a starter CLAUDE.md from the repository; the memory guide says that if one already exists, it suggests improvements instead of overwriting it.

Plan When a Change Can Move Your Eval Numbers

A one-line change to the dedup key between your train and test splits can move every number in the results table.

Side Questions Stay Out of the History, Not Off the Bill

In a long agent session, every turn you add stays in the context the agent reasons from for the rest of the run.

Resume and Rewind Restore the Session, Not Your Experiment

Claude Code has two ways back into earlier work, and neither returns the repo to the state your results came from. Resume brings back the conversation.

Pipe the Failing Run's Log Instead of Describing It

A training run died overnight at step 41,000. Your paraphrase of the traceback ("something about shapes") is less useful than the traceback.

Measure a Model Reviewer's Recall and Precision Before You Trust It

Claude Code ships /code-review as a bundled skill. The commands reference says it reviews "the current diff, or a PR number, branch, or path you pass, for…

Test the Boundary Your Mock Removes From an Eval Harness

A mocked model client returns exactly what you told it to: well-formed JSON, every field present, no errors.

Reproduce Your Results Table From a Fresh Clone

A results table that only reproduces on your laptop depends on something you never wrote down: an exported variable, a dataset in your home directory, a…

Check Which AGENTS.md Files Codex Loaded Before Adding Rules

Before a model can follow a rule, the rule has to be in its context. Chapter 21 treats that context as a budget assembled from parts; for Codex, the…

Teach Claude That a Blank Eval Score Is Not a Zero

In an eval results table, a blank score can mean the run timed out, the API returned an error or the grader crashed.

Preview a Mapping Before the Agent Renames Your Run Directories

Renaming 300 run directories into a clean modellrseed scheme is easy to ask for and hard to undo when two runs collapse into one name. lr=1e-3 and lr=0.001…

Check That Cleaning an Eval CSV Keeps Its Meaning

An eval results CSV carries meaning a cleanup script can erase. Item ID 00127 is a string, not the number 127.

Let Claude Interview You Before You Build an Eval

"Can you build an eval for tool-use regressions?" sounds like one task. It hides a dozen decisions: which model and snapshot, what counts as a pass, how many…

Point the Agent at the Exact File, Revision and Failing Case

Your eval reports 0.71 on a benchmark and the paper reports 0.78. You already suspect the answer normalization in eval/metrics.py, and you have one case where…