Worked examples and arguments about AI systems: how they work, implementation choices, evaluation, costs, failures and what they change about organizations.
Build a workflow around your coding agent. The dark software factory workshop connects the explanations to one runnable application, acceptance checks and a release decision.
Archive
70 articles, newest first.
49–60 of 70 articles
Newest first
A checkpoint records what your job intended, not what the outside world did. Split pure steps from side effects, record an uncertain send before making it, and choose a checkpoint interval from the crash rate.
Similarity search ranks by meaning, not currency, so a note that used to be true can outrank the one that is. Measure it as stale@1 and recall@k of the current note, then retire by source revision.
Model routing is an evaluation problem. Derive cost per accepted outcome as cost per attempt divided by acceptance rate, put intervals on it, and decide each task family on your own graded cases instead of a leaderboard.
An agent that serves many people through one credential is a confused deputy waiting to happen. Keep caller and credential separate, check the caller before every call, and test every caller-resource pair.
A stateless MCP transport lets any server instance take any request, and it leaves job state and authorization to your application. Keep both outside the connection, and check the caller on every protected request.
A new hostname does not mean new authority. Test an agent release in a preview whose credentials cannot launch real jobs, and keep the unit-test result, the credential policy and the receiver-side log as separate evidence.
A benchmark score is an estimate from a sample of tasks, graded by an oracle that can be wrong, on data a model may have seen. Put error bars on it and test contamination and oracle validity.
Save a failed agent trial as a regression case that keeps the observation, the hypotheses and the expected recovery apart, and learn how many reruns it takes to call a flaky failure fixed.
A model's eval plan can pass every schema check and still use the test split, drop cases or overspend. Check it against trusted config, and measure how often valid means wrong.
A handoff for a half-finished sweep or eval should let the receiver check every claim, not trust it. Attach a check to each claim, separate claimed sends from receipts, and see why spot-checking is not enough.
sovereign-agent's research example checks every arXiv ID in the agent's report against what its search tool returned. Read the audit, run it, and measure fabrication rates.
Every station in a dark software factory is a tool an agent calls. Wrap a Python function in a typed tool contract, break it three ways, and record a receipt the factory can rerun: commit, versions, command, output and what was not checked.
Learn
Save your progress with a free account.
Checking sign-in…
Save your progress with a free account.
Pick up where you left off.
Continue learning at my deskCookie duty
The one he did not eat is a measurement cookie. Say yes and the site loads Google Analytics (GA4), which stores an identifier in your browser and lets us count which lessons get read.
Change your mind any time with Privacy choices in the footer. Read the Privacy Policy
Optional · newsletter
That is saved and settled. Separately, and this is marketing rather than measurement: Rod writes a weekly email — one lesson, one field note — sent through Kit. It has nothing to do with cookies, and skipping it costs you nothing.
We will send a confirmation email. Click the link in it and the next lesson finds you.
