All sections
Latest
Everything published, newest first.
The log
Entries 21–40 of 142 · by original publication date
1 entry
2 entries
Test the Date Cutoff in Your Contamination Filter
One common way to reduce contamination in a coding or math eval is a date filter: keep only problems published after the model's training cutoff.Quick thoughtclaude-codeschedulingTurn a Grading Policy Into a Test Table, Then Mutate the Grader
Write boundary cases for an eval grader from its policy, compute expected results by hand, then break the grader on purpose to prove every row can fail. Seven mutants, seven kills.9 min readevaluationmutation testing
2 entries
Ask Claude to Run the Data Prep Twice Before You Trust the Row Count
After a crash, someone reruns the eval-set builder. If the script appends instead of replacing, the test set now holds every example twice, and nothing fails.Quick thoughtclaude-codereliabilityKeep Enough Trace Detail to Grade and Reproduce an Agent Run
A truncated agent transcript can look like a clean success. Record what a trial needs for grading and replay, mark truncation explicitly, and size how many transcripts to read.9 min readagent transcriptstrajectory review
2 entries
Capture Agent Trajectories for Evals Without Copying User Data
Agent traces are the raw material for evals, and production traces carry user data. Capture structure by default, keep content in a restricted store, test every serialization path with canaries, and bound leaks by sampling.8 min readagent tracesagent-evaluation
Measure Whether a Citation Checker Can Tell Support From Topic
A research agent writes a related-work paragraph: "Method X improves GSM8K accuracy by 4 points [3]." Source [3] is about Method X, and it mentions GSM8K.Quick thoughtopenaievaluation1 entry
Cancel an Eval Run or Agent Job Without Misreporting What Ran
A canceled eval run still has requests in flight, bills for them, and leaves a biased partial score. Name the states, count the in-flight work with Little's law, and report only what the evidence supports.10 min readagent cancellationevaluation runs
4 entries
Ask Your LLM Grader for the Smallest Output That Still Carries Evidence
An LLM grader in an eval pipeline often writes a paragraph of rationale for every item. Across a few thousand items, that paragraph is most of your wait, and…Quick thoughtopenailatencyRecheck What a Human Approved at the Moment the Agent Acts
An approval is a check; the agent's action is the use. Close the time-of-check-to-time-of-use gap: bind approval to exact state, estimate how often it goes stale, and make the final check atomic.9 min readhuman approvalTOCTOU
What an AI Agent Remembers After a Restart
After a restart, an agent remembers what your application stored and nothing else. Run a short Python example on Colab that loses its conversation and keeps a corrected preference.7 min readagent-memoryagent state design
What Frontier Labs Ask For, Counted and Read Closely
On October 2, 2026, the jobs board counted 1,837 open technical roles at 49 labs. Here is what research and applied postings mention most, then fifteen postings read in the labs' own words.12 min readfrontier lab careersmember of technical staffUpdated
2 entries
Time Each Stage of an Agent Run Before You Optimize It
An eval sweep that takes all night limits how many ablations you can run in a week, and "why is this agent slow?" is a fair interview question for an…Quick thoughtopenailatencyWhen a Tool Fails, Return a Typed Error, Not an Empty List
An experiment-tracker timeout that returns [] tells your agent no prior runs exist. Keep failure and emptiness apart with typed results, a mutation test, and a protocol to measure the damage.10 min readtool useerror handling
2 entries
Split Agent Tools by Effect, Then Measure the Unsafe-Call Rate
A tool whose name hides a destructive branch makes the dispatcher's job and the model's job harder. Split tools by effect class, then measure merged against split under injection, with intervals.7 min readtool-designagent-evaluation
Use the Batch API When Your Eval Can Wait a Day
A nightly sweep over 10,000 eval items and the debugging session where you rerun five failures have different deadlines.Quick thoughtopenaibatch-processing2 entries
Join Batch Eval Results by custom_id, Never by Line Number
Scoring 5,000 eval items through OpenAI's Batch API costs less than sending them one at a time. The guide lists a 50% discount against the synchronous APIs…Quick thoughtopenaibatch-processingMeasure Agent Latency: Time to First Token, Decode and the p95 Tail
Measured on small open models you can run on Colab: prefill that grows with the prompt, decode at a fixed cost per token, a prefix cache 22 times faster, and a queue that explodes near capacity.12 min readlatencyinference-cost
2 entries
Did Model A Really Beat Model B? Eval Variance, Measured
Measured on 100 AIME problems, ten runs each: single runs of GPT-6 Luna scored anywhere from 88% to 98%. Derive task-level errors, pass@k and pass^k, and paired intervals, then play with them.11 min readagent evaluation varianceevaluation-statisticsUpdated
Put the Rubric First, Then Measure Your Prompt-Cache Hits
Grading 2,000 eval items with the same rubric and the same few-shot examples means sending the same long prefix 2,000 times.Quick thoughtopenaiprompt-cachingContinued on the next page.
To browse rather than follow: topics, all articles and all notes. To learn in order: courses and the book.
