All sections
Latest
Everything published, newest first.
The log
Entries 1–20 of 103
2 entries
Make a Tool an Agent Can Operate: Ten Rules from a Retrofit
Agents lose time on tools built for people: prompts, one exit code for everything, prose output and polling. Build a small CLI that fixes each one, and check it the way an agent will.11 min readagent toolingcommand-line interfaces
1 entry
Review a Month of Coding-Agent Work by Accepted Outcomes
Judge a month of agent work from every attempt, not only the accepted ones: fix the denominator, compare within task families, put error bars on each rate and keep unresolved jobs in the ledger.9 min readagent-evaluationcoding-agents
1 entry
A Coding Agent That Writes Its Own Tests Is Grading Itself
When a coding agent writes both the change and the tests, a passing suite measures the agent's idea of done. Treat it as an eval problem: tests written from the spec, exit codes, rendered measurements and a named human verdict.8 min readcoding-agentsagent-evaluation
2 entries
Check a Codex Fix Against the Original Failing Case
Your eval harness scores cases with an empty reference answer as correct, which inflates accuracy by the share of such cases.Quick thoughtcodexcode-reviewRead an Agent Architecture Diagram by Tracing Every Arrow to Code
A box labeled trusted budget means nothing if its arrow carries a field the model wrote. Trace each arrow in an eval agent's diagram to the line of code that reads it.10 min readagent architecturesystem design
1 entry
Give an Unattended Codex Run a Finish Line You Can Check
"Keep improving the eval harness" has no finish line, so an unattended run can wander until something outside it stops it.Quick thoughtcodexautomation1 entry
Shadow-Test an Agent Release Without Duplicate Writes
An agent release is a tuple of model, prompt, tools and config. Run the candidate in shadow on real inputs with its write authority removed, and derive how many clean episodes it takes to bound what you missed.9 min readshadow deploymentagent releases
1 entry
Keep API Keys Out of a Codex Skill Your Team Shares
A Codex skill that launches an eval run against a hosted model needs an API key. The skill itself should never contain one.Quick thoughtcodexskills1 entry
Use a Circuit Breaker When Retries Multiply an Outage
With r retries at each of d layers, one request can become (r + 1)^d sends, and the multiplier peaks exactly when the dependency is down. Derive it, simulate it, and see what a shared breaker changes.12 min readcircuit breakerretry amplification
1 entry
Reserve an Agent's Spending Before Each Request, Not After
A counter in one process cannot stop eight workers from spending the same last dollar. Bound each call's maximum charge, reserve it atomically in a shared ledger, settle from real usage, and test the race.12 min readagent cost controlinference-cost
1 entry
Roll Back an Agent Release as a Whole Tuple, Not One File
Reverting the prompt alone can leave an old prompt talking to a new tool. Treat model, prompt, tools and config as one release tuple, roll back to a tuple you evaluated, and keep irreversible effects in their own ledger.7 min readagent releasesrollback
2 entries
Count Every Row Before You Trust a Dataset Pull
You ask Claude Code to pull last month's eval runs from your tracker through an MCP server, or to page through an API for a dataset.Quick thoughtclaude-codedata-qualityGive Every Agent Memory an Owner, a Source Revision and an Expiry Rule
A stored fact goes stale when time passes or when its source changes. Record who owns it and which revision it came from, select the current fact by rule rather than similarity, and escalate conflicts.7 min readagent-memorymemory-lifecycle
2 entries
Check What a Citation Proves: Provenance Versus Entailment
An exact, byte-matched quote proves where text came from, not that it supports the claim. Test provenance and entailment separately, and compare exact match, NLI and an LLM judge.8 min readcitation verificationresearch agents
Test Your Results Scripts on Awkward Run Names
A results script that aggregates runs/.jsonl works on clean names. Sweeps don't always produce clean names: lr 3e-4.jsonl, a file that starts with a dash, a…Quick thoughtclaude-codefile-operations2 entries
Find the Failing RAG Stage Before You Change the Prompt
Measured on SciFact with small open models: attribute each miss to retrieval, reranking, prompt assembly or generation, then change only the stage that failed. A targeted prompt fix took 107 correct to 151.10 min readRAG retrieval evaluationrecall-at-k
Run It Logged Out Before You Polish the Plots
A notebook can produce clean figures for you and fail on its first cell for anyone else. The difference is usually state you carry and a new reader does not…Quick thoughtclaude-codeauthentication1 entry
Test the Date Cutoff in Your Contamination Filter
One common way to reduce contamination in a coding or math eval is a date filter: keep only problems published after the model's training cutoff.Quick thoughtclaude-codescheduling1 entry
Ask Claude to Run the Data Prep Twice Before You Trust the Row Count
After a crash, someone reruns the eval-set builder. If the script appends instead of replacing, the test set now holds every example twice, and nothing fails.Quick thoughtclaude-codereliabilityContinued on the next page.
To browse rather than follow: topics, all articles and all notes. To learn in order: courses and the book.
