All sections
Latest
Everything published, newest first.
The log
Entries 1–20 of 40
1 entry
2 entries
Measure Human Agreement Before You Trust an LLM Judge
An LLM judge cannot give a stable quality signal if the people who wrote the rubric disagree about which model outputs should pass.Quick thoughtopenaievaluationWrite the Acceptance Spec Before You Build an Eval Judge
An LLM judge can only grade against a written, versioned policy. Name the owner, the rubric, the labeled set and the pass rule before you build it, and test the rubric on people first.9 min readAI evaluationLLM-as-judge
2 entries
A Job ID Is Not a Result: Check the Output Before Marking It Done
A scheduler's job ID, like a deploy hook's 200, says a request was accepted. A five-state model and a small check that marks work done only when the results file exists, from the right checkpoint, with every row.9 min readexperiment pipelinesevaluation infrastructure
Review a Model Upgrade: Count the Regressions
A new model snapshot can raise your eval score and still break items that used to pass. OpenAI's prompt engineering guide says it directly: "Even different…Quick thoughtopenaievaluation1 entry
A Subagent's Report Is Not a Handoff Until the Receiver Can Act
A subagent's report can arrive while everything it learned stays behind. Define handoff acceptance by what the receiver can check and do next, whether it is an orchestrator or a worker resuming a crashed eval run.9 min readhandoffsmulti-agent systems
1 entry
Measure Whether One Tool Schema Works Across Model Providers
The same JSON Schema can be offered to two providers and still produce different calls, costs and refusals. Keep one strict contract, write thin adapters, and run a paired experiment before you call the tool portable.8 min readtool-callingmodel providers
1 entry
Test an Agent's Fallback Path Against the Same Acceptance Bar
A fallback model runs rarely, so production traffic barely measures it and the aggregate hides it. Derive why, then force failover on your eval set and grade the fallback against the primary's contract.8 min readfallback modelsagent-evaluation
1 entry
Test the Boundary Your Mock Removes From an Eval Harness
A mocked model client returns exactly what you told it to: well-formed JSON, every field present, no errors.Quick thoughtclaude-codetesting1 entry
Pin the Environment Before You Try to Reproduce a Result
Two people can run the same eval script on different library versions and argue about a number neither can reproduce.Quick thoughtclaude-codedependencies1 entry
Reproduce Your Results Table From a Fresh Clone
A results table that only reproduces on your laptop depends on something you never wrote down: an exported variable, a dataset in your home directory, a…Quick thoughtclaude-codeconfiguration1 entry
Turn a Grading Policy Into a Test Table, Then Mutate the Grader
Write boundary cases for an eval grader from its policy, compute expected results by hand, then break the grader on purpose to prove every row can fail. Seven mutants, seven kills.9 min readevaluationmutation testing
2 entries
Capture Agent Trajectories for Evals Without Copying User Data
Agent traces are the raw material for evals, and production traces carry user data. Capture structure by default, keep content in a restricted store, test every serialization path with canaries, and bound leaks by sampling.8 min readagent tracesagent-evaluation
Measure Whether a Citation Checker Can Tell Support From Topic
A research agent writes a related-work paragraph: "Method X improves GSM8K accuracy by 4 points [3]." Source [3] is about Method X, and it mentions GSM8K.Quick thoughtopenaievaluation1 entry
Cancel an Eval Run or Agent Job Without Misreporting What Ran
A canceled eval run still has requests in flight, bills for them, and leaves a biased partial score. Name the states, count the in-flight work with Little's law, and report only what the evidence supports.10 min readagent cancellationevaluation runs
2 entries
Ask Your LLM Grader for the Smallest Output That Still Carries Evidence
An LLM grader in an eval pipeline often writes a paragraph of rationale for every item. Across a few thousand items, that paragraph is most of your wait, and…Quick thoughtopenailatencyWhat Frontier Labs Ask For, Counted and Read Closely
On October 2, 2026, the jobs board counted 1,837 open technical roles at 49 labs. Here is what research and applied postings mention most, then fifteen postings read in the labs' own words.12 min readfrontier lab careersmember of technical staffUpdated
1 entry
Time Each Stage of an Agent Run Before You Optimize It
An eval sweep that takes all night limits how many ablations you can run in a week, and "why is this agent slow?" is a fair interview question for an…Quick thoughtopenailatency1 entry
Join Batch Eval Results by custom_id, Never by Line Number
Scoring 5,000 eval items through OpenAI's Batch API costs less than sending them one at a time. The guide lists a 50% discount against the synchronous APIs…Quick thoughtopenaibatch-processing1 entry
Did Model A Really Beat Model B? Eval Variance, Measured
Measured on 100 AIME problems, ten runs each: single runs of GPT-6 Luna scored anywhere from 88% to 98%. Derive task-level errors, pass@k and pass^k, and paired intervals, then play with them.11 min readagent evaluation varianceevaluation-statisticsUpdated
Continued on the next page.
To browse rather than follow: topics, all articles and all notes. To learn in order: courses and the book.
