All sections
Latest
Everything published, newest first.
The log
Entries 41–60 of 103
11 entries
Measure Coding-Agent Uplift Honestly: Case Mix and Simpson's Paradox
A coding agent can look faster overall while being slower on every kind of task. Count the whole task, hold the case mix fixed, and see how Simpson's paradox produces a false speedup.8 min readagent-evaluationcoding-agents
Measure Stale Agent Memory as Recall@k, Then Retire It by Source Revision
Similarity search ranks by meaning, not currency, so a note that used to be true can outrank the one that is. Measure it as stale@1 and recall@k of the current note, then retire by source revision.8 min readagent-memoryretrieval-evaluation
Preview an Agent Change Without Handing It Production Keys
A new hostname does not mean new authority. Test an agent release in a preview whose credentials cannot launch real jobs, and keep the unit-test result, the credential policy and the receiver-side log as separate evidence.7 min readagent releasespreview deployments
Reject an MCP Token That Was Issued for a Different Server
A signed, unexpired, correctly scoped token can still be meant for another server. Check audience, expiry and scope as separate predicates before the handler runs, and test each one with a positive control.7 min readmcpoauth
Resume a Crashed Agent Job or Eval Run Without Repeating Its Side Effects
A checkpoint records what your job intended, not what the outside world did. Split pure steps from side effects, record an uncertain send before making it, and choose a checkpoint interval from the crash rate.10 min readdurable agent executioncrash recovery
Test Whether a Prior Verdict Anchors Your AI Reviewer
Measured on small open models: one line saying a prior reviewer passed the code cut bug detection from 51% to 27%. Seal the reviewer's request, and measure anchoring with a paired test.8 min readAI revieweranchoring bias
Test Whether Extra Context Helps Before You Buy a Longer Window
A bigger context window does not make every passage relevant. OpenAI's prompt engineering guide describes adding context as a way to give the model data it…Quick thoughtopenaipromptingTurn One Failed Agent Run Into a Regression Eval Case
Save a failed agent trial as a regression case that keeps the observation, the hypotheses and the expected recovery apart, and learn how many reruns it takes to call a flaky failure fixed.9 min readagent-evaluationregression testing
Where State and Identity Live Behind a Stateless MCP Server
A stateless MCP transport lets any server instance take any request, and it leaves job state and authorization to your application. Keep both outside the connection, and check the caller on every protected request.8 min readMCPagent infrastructureUpdated
Write a Handoff the Next Engineer or Agent Can Verify
A handoff for a half-finished sweep or eval should let the receiver check every claim, not trust it. Attach a check to each claim, separate claimed sends from receipts, and see why spot-checking is not enough.9 min readagent handoffresearch engineering
1 entry
Ask Codex to Trace a Generated Results Table to Its Source
The results table in your paper repo shows 71.4 where the logs say 74.1. Editing the table by hand fixes the PDF until the next regeneration, and until then…Quick thoughtcodexcode-review1 entry
Turn a Repeated Results Check Into a Tested Codex Skill
Every time a sweep finishes, you paste the same instructions: check that each results file has the required fields, that no item id appears twice, and that…Quick thoughtcodexskills1 entry
Gate a Codex Step on Its Status, Its Schema and Its Numbers
A pipeline of five steps that each pass a bad artifact along 5% of the time delivers a clean result only 0.95⁵ ≈ 0.77 of the time.Quick thoughtcodexautomation1 entry
Pilot a Codex Migration on Two Files, Then Diff the Requests
You need to move 40 experiment configs to a new format: a renamed field, a new default, a changed prompt-template path.Quick thoughtcodextesting1 entry
Give a Codex Skill One Request It Must Refuse: Publishing
A skill that builds a results table from eval logs looks excellent on the request it was written for.Quick thoughtcodexskills1 entry
Give Codex Write Access to the Repo Without Giving It the Network
Fixing a bug in an eval script needs write access to the repository. It does not need the network. But if the script's tests call a paid model API, or…Quick thoughtcodexpermissions1 entry
Write Review Rules Around Your Experiment's Invariants
"Review carefully" gives a code reviewer nothing to prioritize. An invariant does. In research code, the invariants are the rules that keep a number honest: -…Quick thoughtcodexcode-review1 entry
Pin the Working Directory of a Scheduled Codex Run
A reviewer asks where a number came from. The answer you want is a commit and a command. A nightly eval that runs Codex from cron or CI starts wherever the…Quick thoughtcodexautomationUpdated1 entry
Parse Codex's Final Result, Not Its Progress Stream
A sweep driver calls codex exec once per config and builds a results table. If it parses everything the command prints, two things go wrong.Quick thoughtcodexautomationContinued on the next page.
To browse rather than follow: topics, all articles and all notes. To learn in order: courses and the book.
