All sections
Latest
Everything published, newest first.
The log
Entries 21–40 of 56
2 entries
What an AI Agent Remembers After a Restart
After a restart, an agent remembers what your application stored and nothing else. Run a short Python example on Colab that loses its conversation and keeps a corrected preference.7 min readagent-memoryagent state design
1 entry
When a Tool Fails, Return a Typed Error, Not an Empty List
An experiment-tracker timeout that returns [] tells your agent no prior runs exist. Keep failure and emptiness apart with typed results, a mutation test, and a protocol to measure the damage.10 min readtool useerror handling
1 entry
Split Agent Tools by Effect, Then Measure the Unsafe-Call Rate
A tool whose name hides a destructive branch makes the dispatcher's job and the model's job harder. Split tools by effect class, then measure merged against split under injection, with intervals.7 min readtool-designagent-evaluation
1 entry
Measure Agent Latency: Time to First Token, Decode and the p95 Tail
Measured on small open models you can run on Colab: prefill that grows with the prompt, decode at a fixed cost per token, a prefix cache 22 times faster, and a queue that explodes near capacity.12 min readlatencyinference-cost
1 entry
Did Model A Really Beat Model B? Eval Variance, Measured
Measured on 100 AIME problems, ten runs each: single runs of GPT-6 Luna scored anywhere from 88% to 98%. Derive task-level errors, pass@k and pass^k, and paired intervals, then play with them.11 min readagent evaluation varianceevaluation-statisticsUpdated
1 entry
Bound an Agent's Repair Loop by Attempts, Time and Side Effects
A repair loop needs three separate limits: attempts, a deadline, and external work. Derive what each extra repair buys, see why decode time sets the real cap, and measure repair success with an error bar.10 min readLLM repair loopstructured-output
1 entry
Key a Cached Answer to Its Reader's Current Access, Not Just the Query
An answer cache keyed on the query serves a revoked reader their old answer. Add an authorization revision to the key, recheck before release, test the race, and count what per-reader keys cost in tokens.7 min readanswer-cachingpermission-aware-retrieval
1 entry
Measure Prompt Injection as an Attack-Success Rate, Then Bound It
Measured on four small open models you can run on Colab: attack success from 10% to 61%, and a permission gate that let none through. Score it in code and bound it with the right sampling unit.12 min readprompt-injectionadversarial-evaluation
1 entry
Write Repair Feedback That Fixes Format Errors and Stops on Policy
Send a model the field and error code it needs to repair a type error, stop on policy errors, and measure how much feedback detail actually improves repair success.10 min readstructured-outputrepair loops
1 entry
Keep the Budget Out of the Model's Output, Even When a README Asks
An injected line in a repo tells your eval agent the sweep is pre-approved for $5,000. Close the plan schema so no model output can set a budget, and test four override shapes.8 min readprompt-injectionstructured-output
1 entry
Turn a Notebook Cell Into a Contract Your Eval Harness Can Call
A notebook cell that parses one judge verdict is not yet a function a harness can call. Declare accepted inputs, rejected inputs and a stable output, then run eight cases through every entry point.9 min readresearch engineeringPydantic
8 entries
Build an Agent Eval Environment That Resists Reward Hacking
An eval or RL task is an input, an environment and a hidden grader. Freeze the environment, keep the oracle away from the agent, and prove the grader with null, cheating and gold agents.9 min readagent-evaluationRL environments
Calibrate an LLM Judge Before You Trust Its Score
Measured on Claude: asked for a one-word verdict, Haiku 4.5 called 74% of 500 math answers wrong when 52% were. A judge is two recalls, not one score. Derive them, their intervals and the correction, then play with them.10 min readLLM-as-judgeevaluation-statistics
Check the Caller's Authority, Not the Agent's Service Account
An agent that serves many people through one credential is a confused deputy waiting to happen. Keep caller and credential separate, check the caller before every call, and test every caller-resource pair.7 min readconfused-deputyagent-security
Design a Prompt-Caching Experiment That Measures the Right Thing
Derive when prompt caching pays from Anthropic's write premium and read discount, then run a cold, warm and changed-prefix protocol that keeps cost, context and latency apart.10 min readprompt-cachinginference-cost
Filter Retrieval by Permission Before Anything Reaches the Prompt
Permission-aware retrieval keeps unauthorized passages out of the ranked candidates, not just out of the answer. Test it on the prompt bytes, and measure what post-filtering costs in recall@k.8 min readpermission-aware-retrievalrag
Give Model Output a Strict Type Boundary Before It Reaches Your Eval
In lax mode, Pydantic turns a judge's JSON true into a score of 1. Decide which conversions each field may make, test seven values, and replay stored verdicts before you switch.9 min readstructured-outputPydantic
Keep an Agent's Approved Proposal Separate From the Action It Takes
A research agent's validated sweep is a proposal, not a submitted job. Build the boundary between them, bind approval to exact code and config, and measure what it catches.10 min readagent side effectshuman approval
Make an Agent's Retried Write Land Exactly Once
A retried write to a results table or tracker can silently double-count an eval. Name each intended effect with a stable key, choose writes that are safe to repeat, and size the key so it cannot collide.10 min readidempotencyagent tool calls
Continued on the next page.
To browse rather than follow: topics, all articles and all notes. To learn in order: courses and the book.
