All sections
Latest
Everything published, newest first.
The log
Entries 1–20 of 37
2 entries
Make a Tool an Agent Can Operate: Ten Rules from a Retrofit
Agents lose time on tools built for people: prompts, one exit code for everything, prose output and polling. Build a small CLI that fixes each one, and check it the way an agent will.11 min readagent toolingcommand-line interfaces
1 entry
A Coding Agent That Writes Its Own Tests Is Grading Itself
When a coding agent writes both the change and the tests, a passing suite measures the agent's idea of done. Treat it as an eval problem: tests written from the spec, exit codes, rendered measurements and a named human verdict.8 min readcoding-agentsagent-evaluation
1 entry
Read an Agent Architecture Diagram by Tracing Every Arrow to Code
A box labeled trusted budget means nothing if its arrow carries a field the model wrote. Trace each arrow in an eval agent's diagram to the line of code that reads it.10 min readagent architecturesystem design
1 entry
Shadow-Test an Agent Release Without Duplicate Writes
An agent release is a tuple of model, prompt, tools and config. Run the candidate in shadow on real inputs with its write authority removed, and derive how many clean episodes it takes to bound what you missed.9 min readshadow deploymentagent releases
1 entry
Use a Circuit Breaker When Retries Multiply an Outage
With r retries at each of d layers, one request can become (r + 1)^d sends, and the multiplier peaks exactly when the dependency is down. Derive it, simulate it, and see what a shared breaker changes.12 min readcircuit breakerretry amplification
1 entry
Reserve an Agent's Spending Before Each Request, Not After
A counter in one process cannot stop eight workers from spending the same last dollar. Bound each call's maximum charge, reserve it atomically in a shared ledger, settle from real usage, and test the race.12 min readagent cost controlinference-cost
1 entry
Roll Back an Agent Release as a Whole Tuple, Not One File
Reverting the prompt alone can leave an old prompt talking to a new tool. Treat model, prompt, tools and config as one release tuple, roll back to a tuple you evaluated, and keep irreversible effects in their own ledger.7 min readagent releasesrollback
1 entry
Give Every Agent Memory an Owner, a Source Revision and an Expiry Rule
A stored fact goes stale when time passes or when its source changes. Record who owns it and which revision it came from, select the current fact by rule rather than similarity, and escalate conflicts.7 min readagent-memorymemory-lifecycle
1 entry
Check What a Citation Proves: Provenance Versus Entailment
An exact, byte-matched quote proves where text came from, not that it supports the claim. Test provenance and entailment separately, and compare exact match, NLI and an LLM judge.8 min readcitation verificationresearch agents
1 entry
Find the Failing RAG Stage Before You Change the Prompt
Measured on SciFact with small open models: attribute each miss to retrieval, reranking, prompt assembly or generation, then change only the stage that failed. A targeted prompt fix took 107 correct to 151.10 min readRAG retrieval evaluationrecall-at-k
1 entry
Keep Enough Trace Detail to Grade and Reproduce an Agent Run
A truncated agent transcript can look like a clean success. Record what a trial needs for grading and replay, mark truncation explicitly, and size how many transcripts to read.9 min readagent transcriptstrajectory review
2 entries
Recheck What a Human Approved at the Moment the Agent Acts
An approval is a check; the agent's action is the use. Close the time-of-check-to-time-of-use gap: bind approval to exact state, estimate how often it goes stale, and make the final check atomic.9 min readhuman approvalTOCTOU
What an AI Agent Remembers After a Restart
After a restart, an agent remembers what your application stored and nothing else. Run a short Python example on Colab that loses its conversation and keeps a corrected preference.7 min readagent-memoryagent state design
1 entry
When a Tool Fails, Return a Typed Error, Not an Empty List
An experiment-tracker timeout that returns [] tells your agent no prior runs exist. Keep failure and emptiness apart with typed results, a mutation test, and a protocol to measure the damage.10 min readtool useerror handling
1 entry
Split Agent Tools by Effect, Then Measure the Unsafe-Call Rate
A tool whose name hides a destructive branch makes the dispatcher's job and the model's job harder. Split tools by effect class, then measure merged against split under injection, with intervals.7 min readtool-designagent-evaluation
1 entry
Bound an Agent's Repair Loop by Attempts, Time and Side Effects
A repair loop needs three separate limits: attempts, a deadline, and external work. Derive what each extra repair buys, see why decode time sets the real cap, and measure repair success with an error bar.10 min readLLM repair loopstructured-output
1 entry
Key a Cached Answer to Its Reader's Current Access, Not Just the Query
An answer cache keyed on the query serves a revoked reader their old answer. Add an authorization revision to the key, recheck before release, test the race, and count what per-reader keys cost in tokens.7 min readanswer-cachingpermission-aware-retrieval
1 entry
Measure Prompt Injection as an Attack-Success Rate, Then Bound It
Measured on four small open models you can run on Colab: attack success from 10% to 61%, and a permission gate that let none through. Score it in code and bound it with the right sampling unit.12 min readprompt-injectionadversarial-evaluation
1 entry
Keep the Budget Out of the Model's Output, Even When a README Asks
An injected line in a repo tells your eval agent the sweep is pre-approved for $5,000. Close the plan schema so no model output can set a budget, and test four override shapes.8 min readprompt-injectionstructured-output
Continued on the next page.
To browse rather than follow: topics, all articles and all notes. To learn in order: courses and the book.
