All sections
Latest
Everything published, newest first.
The log
Entries 21–40 of 40
1 entry
2 entries
Check That Your Judge Prompt's Examples Agree With Its Rubric
An LLM judge prompt usually has a rubric and a few examples. When an example contradicts the rubric, the model has two versions of the job, and you can't…Quick thoughtopenaipromptingTurn a Notebook Cell Into a Contract Your Eval Harness Can Call
A notebook cell that parses one judge verdict is not yet a function a harness can call. Declare accepted inputs, rejected inputs and a stable output, then run eight cases through every entry point.9 min readresearch engineeringPydantic
7 entries
A Contamination Cutoff Needs a Clock, Not Just a Date
A training cutoff of March 31 is a date without a clock. One post can land on either side of it depending on the time zone. How to pick one clock, size the ambiguous boundary day and keep it out of a held-out split.8 min readevaluationtimezones
Calibrate an LLM Judge Before You Trust Its Score
Measured on Claude: asked for a one-word verdict, Haiku 4.5 called 74% of 500 math answers wrong when 52% were. A judge is two recalls, not one score. Derive them, their intervals and the correction, then play with them.10 min readLLM-as-judgeevaluation-statistics
Catch Duplicate Eval Results Before a Dictionary Hides Them
A retry appends a second result for the same case. Build a dictionary keyed by case ID and the pass rate now depends on row order; keep both rows and the error bar shrinks for no reason. Check the list first.8 min readevaluationdata integrity
Give Model Output a Strict Type Boundary Before It Reaches Your Eval
In lax mode, Pydantic turns a judge's JSON true into a score of 1. Decide which conversions each field may make, test seven values, and replay stored verdicts before you switch.9 min readstructured-outputPydantic
Read a Coding Benchmark Score: Contamination, Validity and Error Bars
A benchmark score is an estimate from a sample of tasks, graded by an oracle that can be wrong, on data a model may have seen. Put error bars on it and test contamination and oracle validity.9 min readbenchmark evaluationcoding-agents
Route Between Models by Cost per Accepted Outcome
Model routing is an evaluation problem. Derive cost per accepted outcome as cost per attempt divided by acceptance rate, put intervals on it, and decide each task family on your own graded cases instead of a leaderboard.9 min readmodel routingagent-evaluation
Why Schema-Valid Model Output Can Still Be Wrong, and How to Check It
A model's eval plan can pass every schema check and still use the test split, drop cases or overspend. Check it against trusted config, and measure how often valid means wrong.11 min readstructured-outputevaluation
1 entry
Record the Model Snapshot in Every Row of Your Results Table
A saved prompt is not an experiment record. The model snapshot, the sampling settings and the eval set can each move the score, and a results table that…Quick thoughtopenaievaluation1 entry
Freeze the Eval Before You Tune the Prompt, Then Hold Some Out
If you decide what "better" means after reading each new answer, prompt tuning will always look like progress. Freeze the evaluation first.Quick thoughtopenaievaluation1 entry
A Valid JSON Object Can Still Hold the Wrong Number
OpenAI's Structured Outputs guarantees that the model's response adheres to the JSON Schema you supply.Quick thoughtopenaistructured-output1 entry
Test a Model on the Hard Slice, and Put an Interval on It
An eval average can look healthy while the model fails the cases that decide whether it is fit for the task.Quick thoughtopenaievaluation1 entry
Find the Change You Never Asked the Agent to Make
You asked an agent to fix an axis label on a results plot. The diff also switched the F1 score from macro to micro averaging, because the agent read it as a…Quick thoughtclaude-codecode-review1 entry
Teach Claude That a Blank Eval Score Is Not a Zero
In an eval results table, a blank score can mean the run timed out, the API returned an error or the grader crashed.Quick thoughtclaude-codedata-quality1 entry
Trace One Failed Eval Item Through the Whole Harness
Your eval run finished, the score dropped four points, and the log has a hundred warnings. The loudest one is rarely the cause.Quick thoughtclaude-codedebugging1 entry
Check That Cleaning an Eval CSV Keeps Its Meaning
An eval results CSV carries meaning a cleanup script can erase. Item ID 00127 is a string, not the number 127.Quick thoughtclaude-codedata-quality1 entry
Clean an Eval Set With a Coding Agent, Then Prove It Dropped Nothing
An eval set merged from three sources has mixed label spellings, near-duplicate prompts and empty references. A coding agent will tidy it. Three counts, computed by your own script, show whether it dropped a row that would fake a gain.8 min readevaluationdata-quality
1 entry
Let Claude Interview You Before You Build an Eval
"Can you build an eval for tool-use regressions?" sounds like one task. It hides a dozen decisions: which model and snapshot, what counts as a pass, how many…Quick thoughtclaude-codepracticeThat is the first entry in the log.
To browse rather than follow: topics, all articles and all notes. To learn in order: courses and the book.
