All sections
Latest
Everything published, newest first.
The log
Entries 41–54 of 54
10 entries
Read a Coding Benchmark Score: Contamination, Validity and Error Bars
A benchmark score is an estimate from a sample of tasks, graded by an oracle that can be wrong, on data a model may have seen. Put error bars on it and test contamination and oracle validity.9 min readbenchmark evaluationcoding-agents
Reject an MCP Token That Was Issued for a Different Server
A signed, unexpired, correctly scoped token can still be meant for another server. Check audience, expiry and scope as separate predicates before the handler runs, and test each one with a positive control.7 min readmcpoauth
Resume a Crashed Agent Job or Eval Run Without Repeating Its Side Effects
A checkpoint records what your job intended, not what the outside world did. Split pure steps from side effects, record an uncertain send before making it, and choose a checkpoint interval from the crash rate.10 min readdurable agent executioncrash recovery
Route Between Models by Cost per Accepted Outcome
Model routing is an evaluation problem. Derive cost per accepted outcome as cost per attempt divided by acceptance rate, put intervals on it, and decide each task family on your own graded cases instead of a leaderboard.9 min readmodel routingagent-evaluation
Test Whether a Prior Verdict Anchors Your AI Reviewer
Measured on small open models: one line saying a prior reviewer passed the code cut bug detection from 51% to 27%. Seal the reviewer's request, and measure anchoring with a paired test.8 min readAI revieweranchoring bias
Turn One Failed Agent Run Into a Regression Eval Case
Save a failed agent trial as a regression case that keeps the observation, the hypotheses and the expected recovery apart, and learn how many reruns it takes to call a flaky failure fixed.9 min readagent-evaluationregression testing
Where State and Identity Live Behind a Stateless MCP Server
A stateless MCP transport lets any server instance take any request, and it leaves job state and authorization to your application. Keep both outside the connection, and check the caller on every protected request.8 min readMCPagent infrastructureUpdated
Why Schema-Valid Model Output Can Still Be Wrong, and How to Check It
A model's eval plan can pass every schema check and still use the test split, drop cases or overspend. Check it against trusted config, and measure how often valid means wrong.11 min readstructured-outputevaluation
Write a Handoff the Next Engineer or Agent Can Verify
A handoff for a half-finished sweep or eval should let the receiver check every claim, not trust it. Attach a check to each claim, separate claimed sends from receipts, and see why spot-checking is not enough.9 min readagent handoffresearch engineering
2 entries
Build One Station of a Dark Factory: A Typed Tool With a Receipt
Every station in a dark software factory is a tool an agent calls. Wrap a Python function in a typed tool contract, break it three ways, and record a receipt the factory can rerun: commit, versions, command, output and what was not checked.9 min readdark-software-factoriespython
Catch an Agent Citing Papers Its Tools Never Returned
sovereign-agent's research example checks every arXiv ID in the agent's report against what its search tool returned. Read the audit, run it, and measure fabrication rates.7 min readagentsevaluation
2 entries
When to Let a Coding Agent Run Unsupervised, Using Public Evidence
Decide how much autonomy a coding agent gets from what the task can be checked against, how cheaply it undoes, and what METR's published studies do and do not show.9 min readclaude-codecoding-agents
Your Chat History Is Not a Lab Notebook: Make Agent Work Resumable
Keep a lab notebook for agent-driven research: the commit, config, data version, results with their error bars and unresolved jobs. Then test whether a fresh session can continue from it alone.7 min readreproducibilityresearch engineering
That is the first entry in the log.
To browse rather than follow: topics, all articles and all notes. To learn in order: courses and the book.
