Latest

Fri, Oct 9, 2026Friday, October 9, 20261 entry

GuideMeasure models

Write the Acceptance Spec Before You Build an Eval Judge

An LLM judge can only grade against a written, versioned policy. Name the owner, the rubric, the labeled set and the pass rule before you build it, and test the rubric on people first.9 min readAI evaluationLLM-as-judge

Sat, Oct 3, 2026Saturday, October 3, 20261 entry

GuideMeasure models

Measure Whether One Tool Schema Works Across Model Providers

The same JSON Schema can be offered to two providers and still produce different calls, costs and refusals. Keep one strict contract, write thin adapters, and run a paired experiment before you call the tool portable.8 min readtool-callingmodel providers

Fri, Oct 2, 2026Friday, October 2, 20261 entry

GuideMeasure models

Test an Agent's Fallback Path Against the Same Acceptance Bar

A fallback model runs rarely, so production traffic barely measures it and the aggregate hides it. Derive why, then force failover on your eval set and grade the fallback against the primary's contract.8 min readfallback modelsagent-evaluation

Fri, Sep 25, 2026Friday, September 25, 20261 entry

GuideMeasure models

Turn a Grading Policy Into a Test Table, Then Mutate the Grader

Write boundary cases for an eval grader from its policy, compute expected results by hand, then break the grader on purpose to prove every row can fail. Seven mutants, seven kills.9 min readevaluationmutation testing

Wed, Sep 23, 2026Wednesday, September 23, 20261 entry

GuideMeasure models

Capture Agent Trajectories for Evals Without Copying User Data

Agent traces are the raw material for evals, and production traces carry user data. Capture structure by default, keep content in a restricted store, test every serialization path with canaries, and bound leaks by sampling.8 min readagent tracesagent-evaluation

Tue, Sep 22, 2026Tuesday, September 22, 20261 entry

GuideMeasure models

Cancel an Eval Run or Agent Job Without Misreporting What Ran

A canceled eval run still has requests in flight, bills for them, and leaves a biased partial score. Name the states, count the in-flight work with Little's law, and report only what the evidence supports.10 min readagent cancellationevaluation runs

Thu, Sep 17, 2026Thursday, September 17, 20261 entry

GuideMeasure models

Did Model A Really Beat Model B? Eval Variance, Measured

Measured on 100 AIME problems, ten runs each: single runs of GPT-6 Luna scored anywhere from 88% to 98%. Derive task-level errors, pass@k and pass^k, and paired intervals, then play with them.11 min readagent evaluation varianceevaluation-statisticsUpdated Oct 5, 2026

Sun, Sep 13, 2026Sunday, September 13, 20261 entry

GuideMeasure models

Write Repair Feedback That Fixes Format Errors and Stops on Policy

Send a model the field and error code it needs to repair a type error, stop on policy errors, and measure how much feedback detail actually improves repair success.10 min readstructured-outputrepair loops

Fri, Sep 11, 2026Friday, September 11, 20261 entry

GuideMeasure models

Turn a Notebook Cell Into a Contract Your Eval Harness Can Call

A notebook cell that parses one judge verdict is not yet a function a harness can call. Declare accepted inputs, rejected inputs and a stable output, then run eight cases through every entry point.9 min readresearch engineeringPydantic

Thu, Sep 10, 2026Thursday, September 10, 20265 entries

GuideMeasure models

Calibrate an LLM Judge Before You Trust Its Score

Measured on Claude: asked for a one-word verdict, Haiku 4.5 called 74% of 500 math answers wrong when 52% were. A judge is two recalls, not one score. Derive them, their intervals and the correction, then play with them.10 min readLLM-as-judgeevaluation-statistics

GuideMeasure models

Give Model Output a Strict Type Boundary Before It Reaches Your Eval

In lax mode, Pydantic turns a judge's JSON true into a score of 1. Decide which conversions each field may make, test seven values, and replay stored verdicts before you switch.9 min readstructured-outputPydantic

GuideMeasure models

Read a Coding Benchmark Score: Contamination, Validity and Error Bars

A benchmark score is an estimate from a sample of tasks, graded by an oracle that can be wrong, on data a model may have seen. Put error bars on it and test contamination and oracle validity.9 min readbenchmark evaluationcoding-agents

GuideMeasure models

Route Between Models by Cost per Accepted Outcome

Model routing is an evaluation problem. Derive cost per accepted outcome as cost per attempt divided by acceptance rate, put intervals on it, and decide each task family on your own graded cases instead of a leaderboard.9 min readmodel routingagent-evaluation

GuideMeasure models

Why Schema-Valid Model Output Can Still Be Wrong, and How to Check It

A model's eval plan can pass every schema check and still use the test split, drop cases or overspend. Check it against trusted config, and measure how often valid means wrong.11 min readstructured-outputevaluation

That is the first entry in the log.

To browse rather than follow: topics, all articles and all notes. To learn in order: courses and the book.