Articles

Calibrate an LLM Judge Before You Trust Its Score

Measured on Claude: asked for a one-word verdict, Haiku 4.5 called 74% of 500 math answers wrong when 52% were. A judge is two recalls, not one score. Derive them, their intervals and the correction, then play with them.

Catch Duplicate Eval Results Before a Dictionary Hides Them

A retry appends a second result for the same case. Build a dictionary keyed by case ID and the pass rate now depends on row order; keep both rows and the error bar shrinks for no reason. Check the list first.

Filter Retrieval by Permission Before Anything Reaches the Prompt

Permission-aware retrieval keeps unauthorized passages out of the ranked candidates, not just out of the answer. Test it on the prompt bytes, and measure what post-filtering costs in recall@k.

Test Whether a Prior Verdict Anchors Your AI Reviewer

Measured on small open models: one line saying a prior reviewer passed the code cut bug detection from 51% to 27%. Seal the reviewer's request, and measure anchoring with a paired test.

A Contamination Cutoff Needs a Clock, Not Just a Date

A training cutoff of March 31 is a date without a clock. One post can land on either side of it depending on the time zone. How to pick one clock, size the ambiguous boundary day and keep it out of a held-out split.

Design a Prompt-Caching Experiment That Measures the Right Thing

Derive when prompt caching pays from Anthropic's write premium and read discount, then run a cold, warm and changed-prefix protocol that keeps cost, context and latency apart.

Cost per Solved Task: Break-Even Pass Rates, pass@k and Routing

A cheaper model can raise what each solved task costs. Derive cost per solved task as cost per attempt over pass rate, find the break-even pass rate, and route between models.

Give Model Output a Strict Type Boundary Before It Reaches Your Eval

In lax mode, Pydantic turns a judge's JSON true into a score of 1. Decide which conversions each field may make, test seven values, and replay stored verdicts before you switch.

Sep 10, 20269 min read

Match Each Agent Failure to Its Own Recovery Policy

A rate limit, a spend cap, a malformed answer and a timed-out write each need a different response. Classify failures from the provider's documented errors, price each retry, and test the policy by injecting faults.

Make an Agent's Retried Write Land Exactly Once

A retried write to a results table or tracker can silently double-count an eval. Name each intended effect with a stable key, choose writes that are safe to repeat, and size the key so it cannot collide.

Measure Coding-Agent Uplift Honestly: Case Mix and Simpson's Paradox

A coding agent can look faster overall while being slower on every kind of task. Count the whole task, hold the case mix fixed, and see how Simpson's paradox produces a false speedup.

Resume a Crashed Agent Job or Eval Run Without Repeating Its Side Effects

A checkpoint records what your job intended, not what the outside world did. Split pure steps from side effects, record an uncertain send before making it, and choose a checkpoint interval from the crash rate.