All sections
Latest
Everything published, newest first.
The log
Entries 1–20 of 20
1 entry
1 entry
Measure Human Agreement Before You Trust an LLM Judge
An LLM judge cannot give a stable quality signal if the people who wrote the rubric disagree about which model outputs should pass.Quick thoughtopenaievaluation1 entry
Review a Model Upgrade: Count the Regressions
A new model snapshot can raise your eval score and still break items that used to pass. OpenAI's prompt engineering guide says it directly: "Even different…Quick thoughtopenaievaluation1 entry
Test the Boundary Your Mock Removes From an Eval Harness
A mocked model client returns exactly what you told it to: well-formed JSON, every field present, no errors.Quick thoughtclaude-codetesting1 entry
Pin the Environment Before You Try to Reproduce a Result
Two people can run the same eval script on different library versions and argue about a number neither can reproduce.Quick thoughtclaude-codedependencies1 entry
Reproduce Your Results Table From a Fresh Clone
A results table that only reproduces on your laptop depends on something you never wrote down: an exported variable, a dataset in your home directory, a…Quick thoughtclaude-codeconfiguration1 entry
Measure Whether a Citation Checker Can Tell Support From Topic
A research agent writes a related-work paragraph: "Method X improves GSM8K accuracy by 4 points [3]." Source [3] is about Method X, and it mentions GSM8K.Quick thoughtopenaievaluation1 entry
Ask Your LLM Grader for the Smallest Output That Still Carries Evidence
An LLM grader in an eval pipeline often writes a paragraph of rationale for every item. Across a few thousand items, that paragraph is most of your wait, and…Quick thoughtopenailatency1 entry
Time Each Stage of an Agent Run Before You Optimize It
An eval sweep that takes all night limits how many ablations you can run in a week, and "why is this agent slow?" is a fair interview question for an…Quick thoughtopenailatency1 entry
Join Batch Eval Results by custom_id, Never by Line Number
Scoring 5,000 eval items through OpenAI's Batch API costs less than sending them one at a time. The guide lists a 50% discount against the synchronous APIs…Quick thoughtopenaibatch-processing1 entry
Check That Your Judge Prompt's Examples Agree With Its Rubric
An LLM judge prompt usually has a rubric and a few examples. When an example contradicts the rubric, the model has two versions of the job, and you can't…Quick thoughtopenaiprompting1 entry
Record the Model Snapshot in Every Row of Your Results Table
A saved prompt is not an experiment record. The model snapshot, the sampling settings and the eval set can each move the score, and a results table that…Quick thoughtopenaievaluation1 entry
Freeze the Eval Before You Tune the Prompt, Then Hold Some Out
If you decide what "better" means after reading each new answer, prompt tuning will always look like progress. Freeze the evaluation first.Quick thoughtopenaievaluation1 entry
A Valid JSON Object Can Still Hold the Wrong Number
OpenAI's Structured Outputs guarantees that the model's response adheres to the JSON Schema you supply.Quick thoughtopenaistructured-output1 entry
Test a Model on the Hard Slice, and Put an Interval on It
An eval average can look healthy while the model fails the cases that decide whether it is fit for the task.Quick thoughtopenaievaluation1 entry
Find the Change You Never Asked the Agent to Make
You asked an agent to fix an axis label on a results plot. The diff also switched the F1 score from macro to micro averaging, because the agent read it as a…Quick thoughtclaude-codecode-review1 entry
Teach Claude That a Blank Eval Score Is Not a Zero
In an eval results table, a blank score can mean the run timed out, the API returned an error or the grader crashed.Quick thoughtclaude-codedata-quality1 entry
Trace One Failed Eval Item Through the Whole Harness
Your eval run finished, the score dropped four points, and the log has a hundred warnings. The loudest one is rarely the cause.Quick thoughtclaude-codedebugging1 entry
Check That Cleaning an Eval CSV Keeps Its Meaning
An eval results CSV carries meaning a cleanup script can erase. Item ID 00127 is a string, not the number 127.Quick thoughtclaude-codedata-quality1 entry
Let Claude Interview You Before You Build an Eval
"Can you build an eval for tool-use regressions?" sounds like one task. It hides a dozen decisions: which model and snapshot, what counts as a pass, how many…Quick thoughtclaude-codepracticeThat is the first entry in the log.
To browse rather than follow: topics, all articles and all notes. To learn in order: courses and the book.
