All sections
Latest
Everything published, newest first.
The log
Entries 1–20 of 77
1 entry
1 entry
Review a Model Upgrade: Count the Regressions
A new model snapshot can raise your eval score and still break items that used to pass. OpenAI's prompt engineering guide says it directly: "Even different…Quick thoughtopenaievaluation1 entry
Check a Codex Fix Against the Original Failing Case
Your eval harness scores cases with an empty reference answer as correct, which inflates accuracy by the share of such cases.Quick thoughtcodexcode-review1 entry
Give an Unattended Codex Run a Finish Line You Can Check
"Keep improving the eval harness" has no finish line, so an unattended run can wander until something outside it stops it.Quick thoughtcodexautomation1 entry
Keep API Keys Out of a Codex Skill Your Team Shares
A Codex skill that launches an eval run against a hosted model needs an API key. The skill itself should never contain one.Quick thoughtcodexskills1 entry
Test the Boundary Your Mock Removes From an Eval Harness
A mocked model client returns exactly what you told it to: well-formed JSON, every field present, no errors.Quick thoughtclaude-codetesting1 entry
Pin the Environment Before You Try to Reproduce a Result
Two people can run the same eval script on different library versions and argue about a number neither can reproduce.Quick thoughtclaude-codedependencies1 entry
Reproduce Your Results Table From a Fresh Clone
A results table that only reproduces on your laptop depends on something you never wrote down: an exported variable, a dataset in your home directory, a…Quick thoughtclaude-codeconfiguration1 entry
Count Every Row Before You Trust a Dataset Pull
You ask Claude Code to pull last month's eval runs from your tracker through an MCP server, or to page through an API for a dataset.Quick thoughtclaude-codedata-quality1 entry
Test Your Results Scripts on Awkward Run Names
A results script that aggregates runs/.jsonl works on clean names. Sweeps don't always produce clean names: lr 3e-4.jsonl, a file that starts with a dash, a…Quick thoughtclaude-codefile-operations1 entry
Run It Logged Out Before You Polish the Plots
A notebook can produce clean figures for you and fail on its first cell for anyone else. The difference is usually state you carry and a new reader does not…Quick thoughtclaude-codeauthentication1 entry
Test the Date Cutoff in Your Contamination Filter
One common way to reduce contamination in a coding or math eval is a date filter: keep only problems published after the model's training cutoff.Quick thoughtclaude-codescheduling1 entry
Ask Claude to Run the Data Prep Twice Before You Trust the Row Count
After a crash, someone reruns the eval-set builder. If the script appends instead of replacing, the test set now holds every example twice, and nothing fails.Quick thoughtclaude-codereliability1 entry
Measure Whether a Citation Checker Can Tell Support From Topic
A research agent writes a related-work paragraph: "Method X improves GSM8K accuracy by 4 points [3]." Source [3] is about Method X, and it mentions GSM8K.Quick thoughtopenaievaluation1 entry
Ask Your LLM Grader for the Smallest Output That Still Carries Evidence
An LLM grader in an eval pipeline often writes a paragraph of rationale for every item. Across a few thousand items, that paragraph is most of your wait, and…Quick thoughtopenailatency1 entry
Time Each Stage of an Agent Run Before You Optimize It
An eval sweep that takes all night limits how many ablations you can run in a week, and "why is this agent slow?" is a fair interview question for an…Quick thoughtopenailatency1 entry
Use the Batch API When Your Eval Can Wait a Day
A nightly sweep over 10,000 eval items and the debugging session where you rerun five failures have different deadlines.Quick thoughtopenaibatch-processing1 entry
Join Batch Eval Results by custom_id, Never by Line Number
Scoring 5,000 eval items through OpenAI's Batch API costs less than sending them one at a time. The guide lists a 50% discount against the synchronous APIs…Quick thoughtopenaibatch-processing1 entry
Put the Rubric First, Then Measure Your Prompt-Cache Hits
Grading 2,000 eval items with the same rubric and the same few-shot examples means sending the same long prefix 2,000 times.Quick thoughtopenaiprompt-caching1 entry
Give a Research Agent's Tool Failures a Shape It Can Act On
Your research agent has a tool that fetches an eval run from the experiment tracker. When the call fails, the model sees whatever string you return.Quick thoughtopenaitool-callingContinued on the next page.
To browse rather than follow: topics, all articles and all notes. To learn in order: courses and the book.
