Worked examples and arguments about AI systems: how they work, implementation choices, evaluation, costs, failures and what they change about organizations.
Build a workflow around your coding agent. The dark software factory workshop connects the explanations to one runnable application, acceptance checks and a release decision.
Archive
70 articles, newest first.
49–60 of 61 articles in Harness Engineering
Clear filters
Similarity search ranks by meaning, not currency, so a note that used to be true can outrank the one that is. Measure it as stale@1 and recall@k of the current note, then retire by source revision.
Model routing is an evaluation problem. Derive cost per accepted outcome as cost per attempt divided by acceptance rate, put intervals on it, and decide each task family on your own graded cases instead of a leaderboard.
An agent that serves many people through one credential is a confused deputy waiting to happen. Keep caller and credential separate, check the caller before every call, and test every caller-resource pair.
A stateless MCP transport lets any server instance take any request, and it leaves job state and authorization to your application. Keep both outside the connection, and check the caller on every protected request.
A new hostname does not mean new authority. Test an agent release in a preview whose credentials cannot launch real jobs, and keep the unit-test result, the credential policy and the receiver-side log as separate evidence.
A benchmark score is an estimate from a sample of tasks, graded by an oracle that can be wrong, on data a model may have seen. Put error bars on it and test contamination and oracle validity.
Save a failed agent trial as a regression case that keeps the observation, the hypotheses and the expected recovery apart, and learn how many reruns it takes to call a flaky failure fixed.
A model's eval plan can pass every schema check and still use the test split, drop cases or overspend. Check it against trusted config, and measure how often valid means wrong.
A handoff for a half-finished sweep or eval should let the receiver check every claim, not trust it. Attach a check to each claim, separate claimed sends from receipts, and see why spot-checking is not enough.
A Claude Code permission rule is a string pattern, not an intent. Write the commands it should stop, run them through a headless session in a throwaway VM, and count the unintended allows before you trust the rule.
Every call in an agent loop resends the whole context, so input grows with the square of the calls. Derive the cost with caching, price the solved task with an interval, with Claude Code as the worked instance.
Decide how much autonomy a coding agent gets from what the task can be checked against, how cheaply it undoes, and what METR's published studies do and do not show.
Learn
Save your progress with a free account.
Checking sign-in…
Save your progress with a free account.
Pick up where you left off.
Continue learning at my deskCookie duty
The one he did not eat is a measurement cookie. Say yes and the site loads Google Analytics (GA4), which stores an identifier in your browser and lets us count which lessons get read.
Change your mind any time with Privacy choices in the footer. Read the Privacy Policy
Optional · newsletter
That is saved and settled. Separately, and this is marketing rather than measurement: Rod writes a weekly email — one lesson, one field note — sent through Kit. It has nothing to do with cookies, and skipping it costs you nothing.
We will send a confirmation email. Click the link in it and the next lesson finds you.
