All sections
Latest
Everything published, newest first.
The log
Entries 81–100 of 167 · by original publication date
20 entries
Catch Duplicate Eval Results Before a Dictionary Hides Them
A retry appends a second result for the same case. Build a dictionary keyed by case ID and the pass rate now depends on row order; keep both rows and the error bar shrinks for no reason. Check the list first.8 min readevaluationdata integrity
Check the Caller's Authority, Not the Agent's Service Account
An agent that serves many people through one credential is a confused deputy waiting to happen. Keep caller and credential separate, check the caller before every call, and test every caller-resource pair.7 min readconfused-deputyagent-security
Cost per Solved Task: Break-Even Pass Rates, pass@k and Routing
A cheaper model can raise what each solved task costs. Derive cost per solved task as cost per attempt over pass rate, find the break-even pass rate, and route between models.8 min readinference-costagent-evaluation
Design a Prompt-Caching Experiment That Measures the Right Thing
Derive when prompt caching pays from Anthropic's write premium and read discount, then run a cold, warm and changed-prefix protocol that keeps cost, context and latency apart.10 min readprompt-cachinginference-cost
Filter Retrieval by Permission Before Anything Reaches the Prompt
Permission-aware retrieval keeps unauthorized passages out of the ranked candidates, not just out of the answer. Test it on the prompt bytes, and measure what post-filtering costs in recall@k.8 min readpermission-aware-retrievalrag
Give Model Output a Strict Type Boundary Before It Reaches Your Eval
In lax mode, Pydantic turns a judge's JSON true into a score of 1. Decide which conversions each field may make, test seven values, and replay stored verdicts before you switch.9 min readstructured-outputPydantic
Keep an Agent's Approved Proposal Separate From the Action It Takes
A research agent's validated sweep is a proposal, not a submitted job. Build the boundary between them, bind approval to exact code and config, and measure what it catches.10 min readagent side effectshuman approval
Make an Agent's Retried Write Land Exactly Once
A retried write to a results table or tracker can silently double-count an eval. Name each intended effect with a stable key, choose writes that are safe to repeat, and size the key so it cannot collide.10 min readidempotencyagent tool calls
Match Each Agent Failure to Its Own Recovery Policy
A rate limit, a spend cap, a malformed answer and a timed-out write each need a different response. Classify failures from the provider's documented errors, price each retry, and test the policy by injecting faults.10 min readagent retry policyerror handling
Measure Coding-Agent Uplift Honestly: Case Mix and Simpson's Paradox
A coding agent can look faster overall while being slower on every kind of task. Count the whole task, hold the case mix fixed, and see how Simpson's paradox produces a false speedup.8 min readagent-evaluationcoding-agents
Measure Stale Agent Memory as Recall@k, Then Retire It by Source Revision
Similarity search ranks by meaning, not currency, so a note that used to be true can outrank the one that is. Measure it as stale@1 and recall@k of the current note, then retire by source revision.8 min readagent-memoryretrieval-evaluation
Preview an Agent Change Without Handing It Production Keys
A new hostname does not mean new authority. Test an agent release in a preview whose credentials cannot launch real jobs, and keep the unit-test result, the credential policy and the receiver-side log as separate evidence.7 min readagent releasespreview deployments
Read a Coding Benchmark Score: Contamination, Validity and Error Bars
A benchmark score is an estimate from a sample of tasks, graded by an oracle that can be wrong, on data a model may have seen. Put error bars on it and test contamination and oracle validity.9 min readbenchmark evaluationcoding-agents
Reject an MCP Token That Was Issued for a Different Server
A signed, unexpired, correctly scoped token can still be meant for another server. Check audience, expiry and scope as separate predicates before the handler runs, and test each one with a positive control.7 min readmcpoauth
Resume a Crashed Agent Job or Eval Run Without Repeating Its Side Effects
A checkpoint records what your job intended, not what the outside world did. Split pure steps from side effects, record an uncertain send before making it, and choose a checkpoint interval from the crash rate.10 min readdurable agent executioncrash recovery
Route Between Models by Cost per Accepted Outcome
Model routing is an evaluation problem. Derive cost per accepted outcome as cost per attempt divided by acceptance rate, put intervals on it, and decide each task family on your own graded cases instead of a leaderboard.9 min readmodel routingagent-evaluation
Test Whether a Prior Verdict Anchors Your AI Reviewer
Measured on small open models: one line saying a prior reviewer passed the code cut bug detection from 51% to 27%. Seal the reviewer's request, and measure anchoring with a paired test.8 min readAI revieweranchoring bias
Test Whether Extra Context Helps Before You Buy a Longer Window
A bigger context window does not make every passage relevant. OpenAI's prompt engineering guide describes adding context as a way to give the model data it…Quick thoughtopenaipromptingTurn One Failed Agent Run Into a Regression Eval Case
Save a failed agent trial as a regression case that keeps the observation, the hypotheses and the expected recovery apart, and learn how many reruns it takes to call a flaky failure fixed.9 min readagent-evaluationregression testing
Continued on the next page.
To browse rather than follow: topics, all articles and all notes. To learn in order: courses and the book.
