Evals and benchmarks

Agentic Coding with Cursor for Research Engineers

Review a Cursor agent's edit to research code the way a lab reviewer would: which split it reads, what the metric computes, and whether the run repeats.

Claude Code for AI Research

Twelve lessons on running Claude Code on research code so every number it reports arrives with a command, a commit, a seed, an n and an error bar, practiced on the book's public evaluation code.

Why Agent Systems Fail, and How a Lab Would Find Out

Why agent systems fail and how to find out before a user does: reliability over steps, a harness failure taxonomy, evidence gates, and evals with error bars.

The 2x2 for Agent Design: Workflows, Agents, and Per-Layer Evals

Twelve lessons on designing agent systems the way an applied-AI team reviews them: who waits, who decides, what each choice costs in reliability, latency and tokens, and the eval loop every layer needs.

Voice Agents, Not Chatbots: Latency, Barge-In, and Evaluation

An elective for applied-AI candidates: budget the latency of a voice turn, engineer turn-taking and barge-in, and evaluate a voice agent with error bars and a measurement protocol you run yourself.

LangGraph for AI Research

Build research pipelines on LangGraph's current API: an eval harness with error bars, a run that survives a crash, human review of a judge's borderline calls, and the arithmetic of reliability and cost.

Chapter 16: Evaluation as measurement: error bars and paired comparisons

Treat an evaluation as a measurement: Wilson intervals, standard errors clustered by case, McNemar's paired test, sample size and unbiased pass@k. Then build a harness that says what it checked.

Chapter 17: Optimizing against an evaluation: the winner's curse and preferences

Derive the winner's curse and the Bradley-Terry model, and measure a prompt search on a real model. Then turn feedback into versioned, evaluated skill changes with activation and rollback.

Chapter 7: In-context learning: why a skill's exact words must be tested

Measure how much a prompt's wording moves a real model: paraphrases, example order and case-sampling noise. Then load bounded, versioned skill files and activate them only on evaluation evidence.

Write the Acceptance Spec Before You Build an Eval Judge

An LLM judge can only grade against a written, versioned policy. Name the owner, the rubric, the labeled set and the pass rule before you build it, and test the rubric on people first.

A Job ID Is Not a Result: Check the Output Before Marking It Done

A scheduler's job ID, like a deploy hook's 200, says a request was accepted. A five-state model and a small check that marks work done only when the results file exists, from the right checkpoint, with every row.

A Subagent's Report Is Not a Handoff Until the Receiver Can Act

A subagent's report can arrive while everything it learned stays behind. Define handoff acceptance by what the receiver can check and do next, whether it is an orchestrator or a worker resuming a crashed eval run.

Measure Whether One Tool Schema Works Across Model Providers

The same JSON Schema can be offered to two providers and still produce different calls, costs and refusals. Keep one strict contract, write thin adapters, and run a paired experiment before you call the tool portable.

Test an Agent's Fallback Path Against the Same Acceptance Bar

A fallback model runs rarely, so production traffic barely measures it and the aggregate hides it. Derive why, then force failover on your eval set and grade the fallback against the primary's contract.

Turn a Grading Policy Into a Test Table, Then Mutate the Grader

Write boundary cases for an eval grader from its policy, compute expected results by hand, then break the grader on purpose to prove every row can fail. Seven mutants, seven kills.

Capture Agent Trajectories for Evals Without Copying User Data

Agent traces are the raw material for evals, and production traces carry user data. Capture structure by default, keep content in a restricted store, test every serialization path with canaries, and bound leaks by sampling.

Cancel an Eval Run or Agent Job Without Misreporting What Ran

A canceled eval run still has requests in flight, bills for them, and leaves a biased partial score. Name the states, count the in-flight work with Little's law, and report only what the evidence supports.

What Frontier Labs Ask For, Counted and Read Closely

On October 2, 2026, the jobs board counted 1,837 open technical roles at 49 labs. Here is what research and applied postings mention most, then fifteen postings read in the labs' own words.

Did Model A Really Beat Model B? Eval Variance, Measured

Measured on 100 AIME problems, ten runs each: single runs of GPT-6 Luna scored anywhere from 88% to 98%. Derive task-level errors, pass@k and pass^k, and paired intervals, then play with them.

Write Repair Feedback That Fixes Format Errors and Stops on Policy

Send a model the field and error code it needs to repair a type error, stop on policy errors, and measure how much feedback detail actually improves repair success.

Turn a Notebook Cell Into a Contract Your Eval Harness Can Call

A notebook cell that parses one judge verdict is not yet a function a harness can call. Declare accepted inputs, rejected inputs and a stable output, then run eight cases through every entry point.

Calibrate an LLM Judge Before You Trust Its Score

Measured on Claude: asked for a one-word verdict, Haiku 4.5 called 74% of 500 math answers wrong when 52% were. A judge is two recalls, not one score. Derive them, their intervals and the correction, then play with them.

Catch Duplicate Eval Results Before a Dictionary Hides Them

A retry appends a second result for the same case. Build a dictionary keyed by case ID and the pass rate now depends on row order; keep both rows and the error bar shrinks for no reason. Check the list first.

A Contamination Cutoff Needs a Clock, Not Just a Date

A training cutoff of March 31 is a date without a clock. One post can land on either side of it depending on the time zone. How to pick one clock, size the ambiguous boundary day and keep it out of a held-out split.

Give Model Output a Strict Type Boundary Before It Reaches Your Eval

In lax mode, Pydantic turns a judge's JSON true into a score of 1. Decide which conversions each field may make, test seven values, and replay stored verdicts before you switch.

Route Between Models by Cost per Accepted Outcome

Model routing is an evaluation problem. Derive cost per accepted outcome as cost per attempt divided by acceptance rate, put intervals on it, and decide each task family on your own graded cases instead of a leaderboard.

Read a Coding Benchmark Score: Contamination, Validity and Error Bars

A benchmark score is an estimate from a sample of tasks, graded by an oracle that can be wrong, on data a model may have seen. Put error bars on it and test contamination and oracle validity.

Why Schema-Valid Model Output Can Still Be Wrong, and How to Check It

A model's eval plan can pass every schema check and still use the test split, drop cases or overspend. Check it against trusted config, and measure how often valid means wrong.

Clean an Eval Set With a Coding Agent, Then Prove It Dropped Nothing

An eval set merged from three sources has mixed label spellings, near-duplicate prompts and empty references. A coding agent will tidy it. Three counts, computed by your own script, show whether it dropped a row that would fake a gain.

Review a Month of Coding-Agent Work by Accepted Outcomes

Judge a month of agent work from every attempt, not only the accepted ones: fix the denominator, compare within task families, put error bars on each rate and keep unresolved jobs in the ledger.

A Coding Agent That Writes Its Own Tests Is Grading Itself

When a coding agent writes both the change and the tests, a passing suite measures the agent's idea of done. Treat it as an eval problem: tests written from the spec, exit codes, rendered measurements and a named human verdict.

Read an Agent Architecture Diagram by Tracing Every Arrow to Code

A box labeled trusted budget means nothing if its arrow carries a field the model wrote. Trace each arrow in an eval agent's diagram to the line of code that reads it.

Roll Back an Agent Release as a Whole Tuple, Not One File

Reverting the prompt alone can leave an old prompt talking to a new tool. Treat model, prompt, tools and config as one release tuple, roll back to a tuple you evaluated, and keep irreversible effects in their own ledger.

Check What a Citation Proves: Provenance Versus Entailment

An exact, byte-matched quote proves where text came from, not that it supports the claim. Test provenance and entailment separately, and compare exact match, NLI and an LLM judge.

Bound an Agent's Repair Loop by Attempts, Time and Side Effects

A repair loop needs three separate limits: attempts, a deadline, and external work. Derive what each extra repair buys, see why decode time sets the real cap, and measure repair success with an error bar.

Measure Prompt Injection as an Attack-Success Rate, Then Bound It

Measured on four small open models you can run on Colab: attack success from 10% to 61%, and a permission gate that let none through. Score it in code and bound it with the right sampling unit.

Measure Whether a Coding Agent Weakens Tests Under Pressure

Coding agents sometimes pass by editing the tests instead of the code. Here is the evidence, the arithmetic, and a protocol you can run to measure how often it happens.

Keep the Budget Out of the Model's Output, Even When a README Asks

An injected line in a repo tells your eval agent the sweep is pre-approved for $5,000. Close the plan schema so no model output can set a budget, and test four override shapes.

What You Still Decide When a Coding Agent Builds the Change

A coding agent can implement a change to your eval runner. It cannot decide what counts as the same request, what a retry may do, or who may change the tests.

Build an Agent Eval Environment That Resists Reward Hacking

An eval or RL task is an input, an environment and a hidden grader. Freeze the environment, keep the oracle away from the agent, and prove the grader with null, cheating and gold agents.

Cost per Solved Task: Break-Even Pass Rates, pass@k and Routing

A cheaper model can raise what each solved task costs. Derive cost per solved task as cost per attempt over pass rate, find the break-even pass rate, and route between models.

Match Each Agent Failure to Its Own Recovery Policy

A rate limit, a spend cap, a malformed answer and a timed-out write each need a different response. Classify failures from the provider's documented errors, price each retry, and test the policy by injecting faults.

Make an Agent's Retried Write Land Exactly Once

A retried write to a results table or tracker can silently double-count an eval. Name each intended effect with a stable key, choose writes that are safe to repeat, and size the key so it cannot collide.

Resume a Crashed Agent Job or Eval Run Without Repeating Its Side Effects

A checkpoint records what your job intended, not what the outside world did. Split pure steps from side effects, record an uncertain send before making it, and choose a checkpoint interval from the crash rate.

Measure Stale Agent Memory as Recall@k, Then Retire It by Source Revision

Similarity search ranks by meaning, not currency, so a note that used to be true can outrank the one that is. Measure it as stale@1 and recall@k of the current note, then retire by source revision.

Where State and Identity Live Behind a Stateless MCP Server

A stateless MCP transport lets any server instance take any request, and it leaves job state and authorization to your application. Keep both outside the connection, and check the caller on every protected request.

Turn One Failed Agent Run Into a Regression Eval Case

Save a failed agent trial as a regression case that keeps the observation, the hypotheses and the expected recovery apart, and learn how many reruns it takes to call a flaky failure fixed.

In a Dark Factory, Checking Is the Real Price of an Answer

As model calls get cheaper, what a dark software factory's output costs comes to be set by review time and acceptance rate, not tokens. The arithmetic, the error bar that decides a route, and the accounting that keeps it honest.

Measure Human Agreement Before You Trust an LLM Judge

An LLM judge cannot give a stable quality signal if the people who wrote the rubric disagree about which model outputs should pass.

Review a Model Upgrade: Count the Regressions

A new model snapshot can raise your eval score and still break items that used to pass. OpenAI's prompt engineering guide says it directly: "Even different…

Test the Boundary Your Mock Removes From an Eval Harness

A mocked model client returns exactly what you told it to: well-formed JSON, every field present, no errors.

Measure Whether a Citation Checker Can Tell Support From Topic

A research agent writes a related-work paragraph: "Method X improves GSM8K accuracy by 4 points [3]." Source [3] is about Method X, and it mentions GSM8K.

Ask Your LLM Grader for the Smallest Output That Still Carries Evidence

An LLM grader in an eval pipeline often writes a paragraph of rationale for every item. Across a few thousand items, that paragraph is most of your wait, and…

Time Each Stage of an Agent Run Before You Optimize It

An eval sweep that takes all night limits how many ablations you can run in a week, and "why is this agent slow?" is a fair interview question for an…

Join Batch Eval Results by custom_id, Never by Line Number

Scoring 5,000 eval items through OpenAI's Batch API costs less than sending them one at a time. The guide lists a 50% discount against the synchronous APIs…

Check That Your Judge Prompt's Examples Agree With Its Rubric

An LLM judge prompt usually has a rubric and a few examples. When an example contradicts the rubric, the model has two versions of the job, and you can't…

Record the Model Snapshot in Every Row of Your Results Table

A saved prompt is not an experiment record. The model snapshot, the sampling settings and the eval set can each move the score, and a results table that…

Freeze the Eval Before You Tune the Prompt, Then Hold Some Out

If you decide what "better" means after reading each new answer, prompt tuning will always look like progress. Freeze the evaluation first.

A Valid JSON Object Can Still Hold the Wrong Number

OpenAI's Structured Outputs guarantees that the model's response adheres to the JSON Schema you supply.

Test a Model on the Hard Slice, and Put an Interval on It

An eval average can look healthy while the model fails the cases that decide whether it is fit for the task.

Find the Change You Never Asked the Agent to Make

You asked an agent to fix an axis label on a results plot. The diff also switched the F1 score from macro to micro averaging, because the agent read it as a…

Teach Claude That a Blank Eval Score Is Not a Zero

In an eval results table, a blank score can mean the run timed out, the API returned an error or the grader crashed.

Trace One Failed Eval Item Through the Whole Harness

Your eval run finished, the score dropped four points, and the log has a hundred warnings. The loudest one is rarely the cause.

Check That Cleaning an Eval CSV Keeps Its Meaning

An eval results CSV carries meaning a cleanup script can erase. Item ID 00127 is a string, not the number 127.

Let Claude Interview You Before You Build an Eval

"Can you build an eval for tool-use regressions?" sounds like one task. It hides a dozen decisions: which model and snapshot, what counts as a pass, how many…

Use the Batch API When Your Eval Can Wait a Day

A nightly sweep over 10,000 eval items and the debugging session where you rerun five failures have different deadlines.

Put the Rubric First, Then Measure Your Prompt-Cache Hits

Grading 2,000 eval items with the same rubric and the same few-shot examples means sending the same long prefix 2,000 times.

Test Whether Extra Context Helps Before You Buy a Longer Window

A bigger context window does not make every passage relevant. OpenAI's prompt engineering guide describes adding context as a way to give the model data it…

Give Claude a Failing Eval Fixture Before You Ask for a Fix

"The accuracy looks too high" gives Claude Code nothing it can check. A three-row fixture does. Suppose the eval set was merged from two shards and some…

Pick a Subagent's Model by Measuring It on Your Own Tasks

The slug says run cheap subagents and reserve Opus for the hard part. That is a reasonable hypothesis, not a rule.

Measure What a Second Model Catches Before Adding It to Review

Measuring one reviewer means its recall on planted bugs and its precision on real findings. This note asks a different question: does a second reviewer catch…

Point the Agent at the Exact File, Revision and Failing Case

Your eval reports 0.71 on a benchmark and the paper reports 0.78. You already suspect the answer normalization in eval/metrics.py, and you have one case where…

Plan When a Change Can Move Your Eval Numbers

A one-line change to the dedup key between your train and test splits can move every number in the results table.

Measure a Model Reviewer's Recall and Precision Before You Trust It

Claude Code ships /code-review as a bundled skill. The commands reference says it reviews "the current diff, or a PR number, branch, or path you pass, for…