All sections
Latest
Everything published, newest first.
The log
Entries 41–60 of 167 · by original publication date
2 entries
Give Every Agent Memory an Owner, a Source Revision and an Expiry Rule
A stored fact goes stale when time passes or when its source changes. Record who owns it and which revision it came from, select the current fact by rule rather than similarity, and escalate conflicts.7 min readagent-memorymemory-lifecycle
2 entries
Check What a Citation Proves: Provenance Versus Entailment
An exact, byte-matched quote proves where text came from, not that it supports the claim. Test provenance and entailment separately, and compare exact match, NLI and an LLM judge.8 min readcitation verificationresearch agents
Test Your Results Scripts on Awkward Run Names
A results script that aggregates runs/.jsonl works on clean names. Sweeps don't always produce clean names: lr 3e-4.jsonl, a file that starts with a dash, a…Quick thoughtclaude-codefile-operations2 entries
Find the Failing RAG Stage Before You Change the Prompt
Measured on SciFact with small open models: attribute each miss to retrieval, reranking, prompt assembly or generation, then change only the stage that failed. A targeted prompt fix took 107 correct to 151.10 min readRAG retrieval evaluationrecall-at-k
Run It Logged Out Before You Polish the Plots
A notebook can produce clean figures for you and fail on its first cell for anyone else. The difference is usually state you carry and a new reader does not…Quick thoughtclaude-codeauthentication2 entries
Test the Date Cutoff in Your Contamination Filter
One common way to reduce contamination in a coding or math eval is a date filter: keep only problems published after the model's training cutoff.Quick thoughtclaude-codeschedulingTurn a Grading Policy Into a Test Table, Then Mutate the Grader
Write boundary cases for an eval grader from its policy, compute expected results by hand, then break the grader on purpose to prove every row can fail. Seven mutants, seven kills.9 min readevaluationmutation testing
2 entries
Ask Claude to Run the Data Prep Twice Before You Trust the Row Count
After a crash, someone reruns the eval-set builder. If the script appends instead of replacing, the test set now holds every example twice, and nothing fails.Quick thoughtclaude-codereliabilityKeep Enough Trace Detail to Grade and Reproduce an Agent Run
A truncated agent transcript can look like a clean success. Record what a trial needs for grading and replay, mark truncation explicitly, and size how many transcripts to read.9 min readagent transcriptstrajectory review
2 entries
Capture Agent Trajectories for Evals Without Copying User Data
Agent traces are the raw material for evals, and production traces carry user data. Capture structure by default, keep content in a restricted store, test every serialization path with canaries, and bound leaks by sampling.8 min readagent tracesagent-evaluation
Measure Whether a Citation Checker Can Tell Support From Topic
A research agent writes a related-work paragraph: "Method X improves GSM8K accuracy by 4 points [3]." Source [3] is about Method X, and it mentions GSM8K.Quick thoughtopenaievaluation1 entry
Cancel an Eval Run or Agent Job Without Misreporting What Ran
A canceled eval run still has requests in flight, bills for them, and leaves a biased partial score. Name the states, count the in-flight work with Little's law, and report only what the evidence supports.10 min readagent cancellationevaluation runs
4 entries
Ask Your LLM Grader for the Smallest Output That Still Carries Evidence
An LLM grader in an eval pipeline often writes a paragraph of rationale for every item. Across a few thousand items, that paragraph is most of your wait, and…Quick thoughtopenailatencyRecheck What a Human Approved at the Moment the Agent Acts
An approval is a check; the agent's action is the use. Close the time-of-check-to-time-of-use gap: bind approval to exact state, estimate how often it goes stale, and make the final check atomic.9 min readhuman approvalTOCTOU
What an AI Agent Remembers After a Restart
After a restart, an agent remembers what your application stored and nothing else. Run a short Python example on Colab that loses its conversation and keeps a corrected preference.7 min readagent-memoryagent state design
What Frontier Labs Ask For, Counted and Read Closely
On October 2, 2026, the jobs board counted 1,837 open technical roles at 49 labs. Here is what research and applied postings mention most, then fifteen postings read in the labs' own words.12 min readfrontier lab careersmember of technical staffUpdated
2 entries
Time Each Stage of an Agent Run Before You Optimize It
An eval sweep that takes all night limits how many ablations you can run in a week, and "why is this agent slow?" is a fair interview question for an…Quick thoughtopenailatencyWhen a Tool Fails, Return a Typed Error, Not an Empty List
An experiment-tracker timeout that returns [] tells your agent no prior runs exist. Keep failure and emptiness apart with typed results, a mutation test, and a protocol to measure the damage.10 min readtool useerror handling
1 entry
Split Agent Tools by Effect, Then Measure the Unsafe-Call Rate
A tool whose name hides a destructive branch makes the dispatcher's job and the model's job harder. Split tools by effect class, then measure merged against split under injection, with intervals.7 min readtool-designagent-evaluation
Continued on the next page.
To browse rather than follow: topics, all articles and all notes. To learn in order: courses and the book.
