Agents and tool use

Agentic Coding with Cursor for Research Engineers

Review a Cursor agent's edit to research code the way a lab reviewer would: which split it reads, what the metric computes, and whether the run repeats.

Why Agent Systems Fail, and How a Lab Would Find Out

Why agent systems fail and how to find out before a user does: reliability over steps, a harness failure taxonomy, evidence gates, and evals with error bars.

Tools, Memory, and Multi-Agent Systems, Measured

Five weeks on the parts that decide whether an agent works: the context it reads, the tools it calls, the strategy it follows and the memory it keeps. Each idea is derived, linked to the book chapter that builds it, and measured on a real model.

The 2x2 for Agent Design: Workflows, Agents, and Per-Layer Evals

Twelve lessons on designing agent systems the way an applied-AI team reviews them: who waits, who decides, what each choice costs in reliability, latency and tokens, and the eval loop every layer needs.

Voice Agents, Not Chatbots: Latency, Barge-In, and Evaluation

An elective for applied-AI candidates: budget the latency of a voice turn, engineer turn-taking and barge-in, and evaluate a voice agent with error bars and a measurement protocol you run yourself.

LangGraph for AI Research

Build research pipelines on LangGraph's current API: an eval harness with error bars, a run that survives a crash, human review of a judge's borderline calls, and the arithmetic of reliability and cost.

Evaluating Coding Agents: Acceptance the Agent Cannot Redefine

Evaluate a coding agent the way a research team would need to: a fixed task set, an acceptance suite the agent cannot edit, and a pass rate with an error bar. Two lessons are available now: the method with the binomial interval derived, then a harness you run that rejects a real agent's defect.

Claude Code for AI Research

Twelve lessons on running Claude Code on research code so every number it reports arrives with a command, a commit, a seed, an n and an error bar, practiced on the book's public evaluation code.

OpenAI Codex for Research Engineers: First Task to Reviewed Change

The check to run before a coding agent touches research code: record a baseline, give the OpenAI Codex CLI one bounded change, read the diff, and compare the outputs yourself. You practice on a small stand-in repository, not an eval repo. Two lessons, both open without an account.

Chapter 2: Structured output and typed tools: what a schema guarantees

Derive what a JSON schema guarantees: constrained decoding masks logits, which is not conditioning on validity. Measure a real model with and without a schema, then build typed tools that validate anyway.

Chapter 3: The agent loop: why reliability compounds, and a loop that stops

Derive why agent reliability compounds as p to the n, how recovery and retries change it, and measure both on a real model. Then build a bounded model and tool loop in Python.

Chapter 4: Build durable state with SQLite

Store Lucy's stock and the events that explain it in SQLite, so a crash, a replay or a half-applied migration can never leave them disagreeing. Measure when a second writer waits.

Chapter 8: Build a durable work inbox and report outbox

Admit each request once per occurrence, refuse work before it is stored when the inbox is full, and finish work with its report in one transaction. Price what resending costs.

Chapter 11: When to ask: calibration, oversight and spending permission

Measure whether a model's confidence could decide which orders need Lucy: calibration, agreement between samples, thinking and pushback. Then bind approval to an exact proposal and its reserved spend.

Chapter 12: Exactly one order: lost replies, retries and idempotency

Derive what retries do when a reply is lost: b/(1 - a - b) duplicate orders, measured over real HTTP. A model re-sends a timed-out order when asked; a stable operation key makes the retry harmless.

Chapter 14: External tools with MCP: the protocol connects, the application decides

Connect Lucy's agent to a tool over MCP. At 64 tools a 1.5b model routed 14 of 24 requests right and Claude 22 to 23, and every tool costs every request tokens. The client decides what runs.

Chapter 18: When a second agent pays: parallelism, errors and bounded delegation

Derive when a second agent pays: Amdahl's law, errors that compound, voting that amplifies both ways, and repeated context, measured on a real model. Then delegate one bounded, read-only task.

Chapter 20: A whole day: reliability, honest reports and readiness

Budget a day's reliability step by step, bound what one passing day proves, and measure whether a model writes a faithful day report. Then run the whole shop day with injected failures.

Chapter 21: Context engineering: a budget for what the model sees

Decide what Lucy's model reads. Measured on small models and Claude: small models lost facts by 16,000 tokens, Claude found all to 150,000, and only pinned records held on every model.

Chapter 13: Slow or dead: leases, fencing and crash recovery

Measure leases on real model turns: false expiry is the latency tail, and detection delay is the lease. Then fence every write by generation and recover from records, not from the model.

Read an Agent Architecture Diagram by Tracing Every Arrow to Code

A box labeled trusted budget means nothing if its arrow carries a field the model wrote. Trace each arrow in an eval agent's diagram to the line of code that reads it.

Shadow-Test an Agent Release Without Duplicate Writes

An agent release is a tuple of model, prompt, tools and config. Run the candidate in shadow on real inputs with its write authority removed, and derive how many clean episodes it takes to bound what you missed.

Use a Circuit Breaker When Retries Multiply an Outage

With r retries at each of d layers, one request can become (r + 1)^d sends, and the multiplier peaks exactly when the dependency is down. Derive it, simulate it, and see what a shared breaker changes.

Reserve an Agent's Spending Before Each Request, Not After

A counter in one process cannot stop eight workers from spending the same last dollar. Bound each call's maximum charge, reserve it atomically in a shared ledger, settle from real usage, and test the race.

Roll Back an Agent Release as a Whole Tuple, Not One File

Reverting the prompt alone can leave an old prompt talking to a new tool. Treat model, prompt, tools and config as one release tuple, roll back to a tuple you evaluated, and keep irreversible effects in their own ledger.

Give Every Agent Memory an Owner, a Source Revision and an Expiry Rule

A stored fact goes stale when time passes or when its source changes. Record who owns it and which revision it came from, select the current fact by rule rather than similarity, and escalate conflicts.

Check What a Citation Proves: Provenance Versus Entailment

An exact, byte-matched quote proves where text came from, not that it supports the claim. Test provenance and entailment separately, and compare exact match, NLI and an LLM judge.

Keep Enough Trace Detail to Grade and Reproduce an Agent Run

A truncated agent transcript can look like a clean success. Record what a trial needs for grading and replay, mark truncation explicitly, and size how many transcripts to read.

Recheck What a Human Approved at the Moment the Agent Acts

An approval is a check; the agent's action is the use. Close the time-of-check-to-time-of-use gap: bind approval to exact state, estimate how often it goes stale, and make the final check atomic.

What an AI Agent Remembers After a Restart

After a restart, an agent remembers what your application stored and nothing else. Run a short Python example on Colab that loses its conversation and keeps a corrected preference.

When a Tool Fails, Return a Typed Error, Not an Empty List

An experiment-tracker timeout that returns [] tells your agent no prior runs exist. Keep failure and emptiness apart with typed results, a mutation test, and a protocol to measure the damage.

Split Agent Tools by Effect, Then Measure the Unsafe-Call Rate

A tool whose name hides a destructive branch makes the dispatcher's job and the model's job harder. Split tools by effect class, then measure merged against split under injection, with intervals.

Bound an Agent's Repair Loop by Attempts, Time and Side Effects

A repair loop needs three separate limits: attempts, a deadline, and external work. Derive what each extra repair buys, see why decode time sets the real cap, and measure repair success with an error bar.

Keep the Budget Out of the Model's Output, Even When a README Asks

An injected line in a repo tells your eval agent the sweep is pre-approved for $5,000. Close the plan schema so no model output can set a budget, and test four override shapes.

Keep an Agent's Approved Proposal Separate From the Action It Takes

A research agent's validated sweep is a proposal, not a submitted job. Build the boundary between them, bind approval to exact code and config, and measure what it catches.

Reject an MCP Token That Was Issued for a Different Server

A signed, unexpired, correctly scoped token can still be meant for another server. Check audience, expiry and scope as separate predicates before the handler runs, and test each one with a positive control.

Build an Agent Eval Environment That Resists Reward Hacking

An eval or RL task is an input, an environment and a hidden grader. Freeze the environment, keep the oracle away from the agent, and prove the grader with null, cheating and gold agents.

Test Whether a Prior Verdict Anchors Your AI Reviewer

Measured on small open models: one line saying a prior reviewer passed the code cut bug detection from 51% to 27%. Seal the reviewer's request, and measure anchoring with a paired test.

Match Each Agent Failure to Its Own Recovery Policy

A rate limit, a spend cap, a malformed answer and a timed-out write each need a different response. Classify failures from the provider's documented errors, price each retry, and test the policy by injecting faults.

Make an Agent's Retried Write Land Exactly Once

A retried write to a results table or tracker can silently double-count an eval. Name each intended effect with a stable key, choose writes that are safe to repeat, and size the key so it cannot collide.

Resume a Crashed Agent Job or Eval Run Without Repeating Its Side Effects

A checkpoint records what your job intended, not what the outside world did. Split pure steps from side effects, record an uncertain send before making it, and choose a checkpoint interval from the crash rate.

Measure Stale Agent Memory as Recall@k, Then Retire It by Source Revision

Similarity search ranks by meaning, not currency, so a note that used to be true can outrank the one that is. Measure it as stale@1 and recall@k of the current note, then retire by source revision.

Check the Caller's Authority, Not the Agent's Service Account

An agent that serves many people through one credential is a confused deputy waiting to happen. Keep caller and credential separate, check the caller before every call, and test every caller-resource pair.

Where State and Identity Live Behind a Stateless MCP Server

A stateless MCP transport lets any server instance take any request, and it leaves job state and authorization to your application. Keep both outside the connection, and check the caller on every protected request.

Preview an Agent Change Without Handing It Production Keys

A new hostname does not mean new authority. Test an agent release in a preview whose credentials cannot launch real jobs, and keep the unit-test result, the credential policy and the receiver-side log as separate evidence.

Turn One Failed Agent Run Into a Regression Eval Case

Save a failed agent trial as a regression case that keeps the observation, the hypotheses and the expected recovery apart, and learn how many reruns it takes to call a flaky failure fixed.

Write a Handoff the Next Engineer or Agent Can Verify

A handoff for a half-finished sweep or eval should let the receiver check every claim, not trust it. Attach a check to each claim, separate claimed sends from receipts, and see why spot-checking is not enough.

Catch an Agent Citing Papers Its Tools Never Returned

sovereign-agent's research example checks every arXiv ID in the agent's report against what its search tool returned. Read the audit, run it, and measure fabrication rates.

A Dark Factory's First Control: Every Claim Carries Its Own Check

Coding agents are fluent at saying they are done. In the dark factory that builds this site, every claim of progress must name the check that would end it, and a gate refuses the ones that do not. What that control catches, and what it cannot.

What a Dark Factory Must Build Itself When Agents Are Cheap

Every station in a dark software factory is a make-or-buy decision. Coase and Williamson worked out the answer decades before agents: buy what is generic, build what is specific to you, and keep the right to decide when the specification runs out.

What an Agent Loop Costs: Quadratic Resend, Cache Reads and Cost per Solved Task

Every call in an agent loop resends the whole context, so input grows with the square of the calls. Derive the cost with caching, price the solved task with an interval, with Claude Code as the worked instance.

Using a Coding Agent on Research Code: Ask Where the Expected Values Came From

An agent can refactor an eval script and pass every test because it rewrote the expected values. A short guide to using Claude Code or any coding agent on research code and take-homes, built around that check.

When to Let a Coding Agent Run Unsupervised, Using Public Evidence

Decide how much autonomy a coding agent gets from what the task can be checked against, how cheaply it undoes, and what METR's published studies do and do not show.

Your Chat History Is Not a Lab Notebook: Make Agent Work Resumable

Keep a lab notebook for agent-driven research: the commit, config, data version, results with their error bars and unresolved jobs. Then test whether a fresh session can continue from it alone.

A Dark Factory Beyond Software, and the Review Arithmetic That Caps It

A dark software factory turns requests into tested software with no person on each change. The same pattern runs any work you can write as code. What caps it is review, not compute: the arithmetic, the control layers, and where it breaks.

A Subagent's Report Is Not a Handoff Until the Receiver Can Act

A subagent's report can arrive while everything it learned stays behind. Define handoff acceptance by what the receiver can check and do next, whether it is an orchestrator or a worker resuming a crashed eval run.

Measure Whether One Tool Schema Works Across Model Providers

The same JSON Schema can be offered to two providers and still produce different calls, costs and refusals. Keep one strict contract, write thin adapters, and run a paired experiment before you call the tool portable.

Test an Agent's Fallback Path Against the Same Acceptance Bar

A fallback model runs rarely, so production traffic barely measures it and the aggregate hides it. Derive why, then force failover on your eval set and grade the fallback against the primary's contract.

Capture Agent Trajectories for Evals Without Copying User Data

Agent traces are the raw material for evals, and production traces carry user data. Capture structure by default, keep content in a restricted store, test every serialization path with canaries, and bound leaks by sampling.

Cancel an Eval Run or Agent Job Without Misreporting What Ran

A canceled eval run still has requests in flight, bills for them, and leaves a biased partial score. Name the states, count the in-flight work with Little's law, and report only what the evidence supports.

Measure Agent Latency: Time to First Token, Decode and the p95 Tail

Measured on small open models you can run on Colab: prefill that grows with the prompt, decode at a fixed cost per token, a prefix cache 22 times faster, and a queue that explodes near capacity.

Measure Prompt Injection as an Attack-Success Rate, Then Bound It

Measured on four small open models you can run on Colab: attack success from 10% to 61%, and a permission gate that let none through. Score it in code and bound it with the right sampling unit.

Write Repair Feedback That Fixes Format Errors and Stops on Policy

Send a model the field and error code it needs to repair a type error, stop on policy errors, and measure how much feedback detail actually improves repair success.

Route Between Models by Cost per Accepted Outcome

Model routing is an evaluation problem. Derive cost per accepted outcome as cost per attempt divided by acceptance rate, put intervals on it, and decide each task family on your own graded cases instead of a leaderboard.

Build One Station of a Dark Factory: A Typed Tool With a Receipt

Every station in a dark software factory is a tool an agent calls. Wrap a Python function in a typed tool contract, break it three ways, and record a receipt the factory can rerun: commit, versions, command, output and what was not checked.

Clean an Eval Set With a Coding Agent, Then Prove It Dropped Nothing

An eval set merged from three sources has mixed label spellings, near-duplicate prompts and empty references. A coding agent will tidy it. Three counts, computed by your own script, show whether it dropped a row that would fake a gain.

Give a Research Agent's Tool Failures a Shape It Can Act On

Your research agent has a tool that fetches an eval run from the experiment tracker. When the call fails, the model sees whatever string you return.

Make 'Propose a Job' and 'Launch a Job' Different Tools

"Propose a training job" and "launch a training job on 64 GPUs" are different operations. So are "draft a cleanup of old checkpoints" and "delete them."

Validate Tool Arguments in Code, Even With Strict Mode

A model's tool call is a request for your code to act. OpenAI's function calling guide lays out the flow: the model returns a tool call, and you "execute code…

Write Review Rules Around Your Experiment's Invariants

"Review carefully" gives a code reviewer nothing to prioritize. An invariant does. In research code, the invariants are the rules that keep a number honest: -…

Check Which AGENTS.md Files Codex Loaded Before Adding Rules

Before a model can follow a rule, the rule has to be in its context. Chapter 21 treats that context as a budget assembled from parts; for Codex, the…

Review an MCP Server's Tools Before an Agent Touches Your Runs

You connect an MCP server for your experiment tracker so the agent can read past runs. The same server may also expose tools that tag, overwrite or delete runs.

Preview a Mapping Before the Agent Renames Your Run Directories

Renaming 300 run directories into a clean modellrseed scheme is easy to ask for and hard to undo when two runs collapse into one name. lr=1e-3 and lr=0.001…

Stop Being the Agent's Verification Loop

You ask if the change is done. The agent says yes. You rerun the eval, and it isn't. The Claude Code best-practices guide names the position you are in…

Point the Agent at the Exact File, Revision and Failing Case

Your eval reports 0.71 on a benchmark and the paper reports 0.78. You already suspect the answer normalization in eval/metrics.py, and you have one case where…

Move to the Agent SDK When You Need to Own the Run

Running an agent by hand from a terminal is how you try an idea. Running it over 200 eval tasks every night is an experiment, and an experiment needs what a…

Time Each Stage of an Agent Run Before You Optimize It

An eval sweep that takes all night limits how many ablations you can run in a week, and "why is this agent slow?" is a fair interview question for an…

Retrieved Text Is Evidence, Not Permission: Measure the Leak Rate

A retrieval step hands the model a document. It contains a fact the task needs and a sentence addressed to the model: ignore your task and call deleterun.

Find the Change You Never Asked the Agent to Make

You asked an agent to fix an axis label on a results plot. The diff also switched the F1 score from macro to micro averaging, because the agent read it as a…