All sections
Latest
Everything published, newest first.
The log
Entries 21–37 of 37
14 entries
Check the Caller's Authority, Not the Agent's Service Account
An agent that serves many people through one credential is a confused deputy waiting to happen. Keep caller and credential separate, check the caller before every call, and test every caller-resource pair.7 min readconfused-deputyagent-security
Filter Retrieval by Permission Before Anything Reaches the Prompt
Permission-aware retrieval keeps unauthorized passages out of the ranked candidates, not just out of the answer. Test it on the prompt bytes, and measure what post-filtering costs in recall@k.8 min readpermission-aware-retrievalrag
Keep an Agent's Approved Proposal Separate From the Action It Takes
A research agent's validated sweep is a proposal, not a submitted job. Build the boundary between them, bind approval to exact code and config, and measure what it catches.10 min readagent side effectshuman approval
Make an Agent's Retried Write Land Exactly Once
A retried write to a results table or tracker can silently double-count an eval. Name each intended effect with a stable key, choose writes that are safe to repeat, and size the key so it cannot collide.10 min readidempotencyagent tool calls
Match Each Agent Failure to Its Own Recovery Policy
A rate limit, a spend cap, a malformed answer and a timed-out write each need a different response. Classify failures from the provider's documented errors, price each retry, and test the policy by injecting faults.10 min readagent retry policyerror handling
Measure Stale Agent Memory as Recall@k, Then Retire It by Source Revision
Similarity search ranks by meaning, not currency, so a note that used to be true can outrank the one that is. Measure it as stale@1 and recall@k of the current note, then retire by source revision.8 min readagent-memoryretrieval-evaluation
Preview an Agent Change Without Handing It Production Keys
A new hostname does not mean new authority. Test an agent release in a preview whose credentials cannot launch real jobs, and keep the unit-test result, the credential policy and the receiver-side log as separate evidence.7 min readagent releasespreview deployments
Reject an MCP Token That Was Issued for a Different Server
A signed, unexpired, correctly scoped token can still be meant for another server. Check audience, expiry and scope as separate predicates before the handler runs, and test each one with a positive control.7 min readmcpoauth
Resume a Crashed Agent Job or Eval Run Without Repeating Its Side Effects
A checkpoint records what your job intended, not what the outside world did. Split pure steps from side effects, record an uncertain send before making it, and choose a checkpoint interval from the crash rate.10 min readdurable agent executioncrash recovery
Test Whether a Prior Verdict Anchors Your AI Reviewer
Measured on small open models: one line saying a prior reviewer passed the code cut bug detection from 51% to 27%. Seal the reviewer's request, and measure anchoring with a paired test.8 min readAI revieweranchoring bias
Turn One Failed Agent Run Into a Regression Eval Case
Save a failed agent trial as a regression case that keeps the observation, the hypotheses and the expected recovery apart, and learn how many reruns it takes to call a flaky failure fixed.9 min readagent-evaluationregression testing
Where State and Identity Live Behind a Stateless MCP Server
A stateless MCP transport lets any server instance take any request, and it leaves job state and authorization to your application. Keep both outside the connection, and check the caller on every protected request.8 min readMCPagent infrastructureUpdated
Write a Handoff the Next Engineer or Agent Can Verify
A handoff for a half-finished sweep or eval should let the receiver check every claim, not trust it. Attach a check to each claim, separate claimed sends from receipts, and see why spot-checking is not enough.9 min readagent handoffresearch engineering
1 entry
Catch an Agent Citing Papers Its Tools Never Returned
sovereign-agent's research example checks every arXiv ID in the agent's report against what its search tool returned. Read the audit, run it, and measure fabrication rates.7 min readagentsevaluation
2 entries
When to Let a Coding Agent Run Unsupervised, Using Public Evidence
Decide how much autonomy a coding agent gets from what the task can be checked against, how cheaply it undoes, and what METR's published studies do and do not show.9 min readclaude-codecoding-agents
Your Chat History Is Not a Lab Notebook: Make Agent Work Resumable
Keep a lab notebook for agent-driven research: the commit, config, data version, results with their error bars and unresolved jobs. Then test whether a fresh session can continue from it alone.7 min readreproducibilityresearch engineering
That is the first entry in the log.
To browse rather than follow: topics, all articles and all notes. To learn in order: courses and the book.
