All sections
Latest
Everything published, newest first.
The log
Entries 21–40 of 167 · by original publication date
1 entry
2 entries
Measure Human Agreement Before You Trust an LLM Judge
An LLM judge cannot give a stable quality signal if the people who wrote the rubric disagree about which model outputs should pass.Quick thoughtopenaievaluationWrite the Acceptance Spec Before You Build an Eval Judge
An LLM judge can only grade against a written, versioned policy. Name the owner, the rubric, the labeled set and the pass rule before you build it, and test the rubric on people first.9 min readAI evaluationLLM-as-judge
2 entries
A Job ID Is Not a Result: Check the Output Before Marking It Done
A scheduler's job ID, like a deploy hook's 200, says a request was accepted. A five-state model and a small check that marks work done only when the results file exists, from the right checkpoint, with every row.9 min readexperiment pipelinesevaluation infrastructure
Review a Model Upgrade: Count the Regressions
A new model snapshot can raise your eval score and still break items that used to pass. OpenAI's prompt engineering guide says it directly: "Even different…Quick thoughtopenaievaluation1 entry
A Coding Agent That Writes Its Own Tests Is Grading Itself
When a coding agent writes both the change and the tests, a passing suite measures the agent's idea of done. Treat it as an eval problem: tests written from the spec, exit codes, rendered measurements and a named human verdict.8 min readcoding-agentsagent-evaluation
2 entries
Check a Codex Fix Against the Original Failing Case
Your eval harness scores cases with an empty reference answer as correct, which inflates accuracy by the share of such cases.Quick thoughtcodexcode-reviewRead an Agent Architecture Diagram by Tracing Every Arrow to Code
A box labeled trusted budget means nothing if its arrow carries a field the model wrote. Trace each arrow in an eval agent's diagram to the line of code that reads it.10 min readagent architecturesystem design
2 entries
A Subagent's Report Is Not a Handoff Until the Receiver Can Act
A subagent's report can arrive while everything it learned stays behind. Define handoff acceptance by what the receiver can check and do next, whether it is an orchestrator or a worker resuming a crashed eval run.9 min readhandoffsmulti-agent systems
Give an Unattended Codex Run a Finish Line You Can Check
"Keep improving the eval harness" has no finish line, so an unattended run can wander until something outside it stops it.Quick thoughtcodexautomation1 entry
Shadow-Test an Agent Release Without Duplicate Writes
An agent release is a tuple of model, prompt, tools and config. Run the candidate in shadow on real inputs with its write authority removed, and derive how many clean episodes it takes to bound what you missed.9 min readshadow deploymentagent releases
2 entries
Keep API Keys Out of a Codex Skill Your Team Shares
A Codex skill that launches an eval run against a hosted model needs an API key. The skill itself should never contain one.Quick thoughtcodexskillsMeasure Whether One Tool Schema Works Across Model Providers
The same JSON Schema can be offered to two providers and still produce different calls, costs and refusals. Keep one strict contract, write thin adapters, and run a paired experiment before you call the tool portable.8 min readtool-callingmodel providers
1 entry
Test an Agent's Fallback Path Against the Same Acceptance Bar
A fallback model runs rarely, so production traffic barely measures it and the aggregate hides it. Derive why, then force failover on your eval set and grade the fallback against the primary's contract.8 min readfallback modelsagent-evaluation
2 entries
Test the Boundary Your Mock Removes From an Eval Harness
A mocked model client returns exactly what you told it to: well-formed JSON, every field present, no errors.Quick thoughtclaude-codetestingUse a Circuit Breaker When Retries Multiply an Outage
With r retries at each of d layers, one request can become (r + 1)^d sends, and the multiplier peaks exactly when the dependency is down. Derive it, simulate it, and see what a shared breaker changes.12 min readcircuit breakerretry amplification
2 entries
Pin the Environment Before You Try to Reproduce a Result
Two people can run the same eval script on different library versions and argue about a number neither can reproduce.Quick thoughtclaude-codedependenciesReserve an Agent's Spending Before Each Request, Not After
A counter in one process cannot stop eight workers from spending the same last dollar. Bound each call's maximum charge, reserve it atomically in a shared ledger, settle from real usage, and test the race.12 min readagent cost controlinference-cost
2 entries
Reproduce Your Results Table From a Fresh Clone
A results table that only reproduces on your laptop depends on something you never wrote down: an exported variable, a dataset in your home directory, a…Quick thoughtclaude-codeconfigurationRoll Back an Agent Release as a Whole Tuple, Not One File
Reverting the prompt alone can leave an old prompt talking to a new tool. Treat model, prompt, tools and config as one release tuple, roll back to a tuple you evaluated, and keep irreversible effects in their own ledger.7 min readagent releasesrollback
Continued on the next page.
To browse rather than follow: topics, all articles and all notes. To learn in order: courses and the book.
