Worked examples and arguments about AI systems: how they work, implementation choices, evaluation, costs, failures and what they change about organizations.
Build a workflow around your coding agent. The dark software factory workshop connects the explanations to one runnable application, acceptance checks and a release decision.
Archive
70 articles, newest first.
1–12 of 70 articles
Newest first
Judge a month of agent work from every attempt, not only the accepted ones: fix the denominator, compare within task families, put error bars on each rate and keep unresolved jobs in the ledger.
An LLM judge can only grade against a written, versioned policy. Name the owner, the rubric, the labeled set and the pass rule before you build it, and test the rubric on people first.
A scheduler's job ID, like a deploy hook's 200, says a request was accepted. A five-state model and a small check that marks work done only when the results file exists, from the right checkpoint, with every row.
When a coding agent writes both the change and the tests, a passing suite measures the agent's idea of done. Treat it as an eval problem: tests written from the spec, exit codes, rendered measurements and a named human verdict.
A box labeled trusted budget means nothing if its arrow carries a field the model wrote. Trace each arrow in an eval agent's diagram to the line of code that reads it.
A subagent's report can arrive while everything it learned stays behind. Define handoff acceptance by what the receiver can check and do next, whether it is an orchestrator or a worker resuming a crashed eval run.
An agent release is a tuple of model, prompt, tools and config. Run the candidate in shadow on real inputs with its write authority removed, and derive how many clean episodes it takes to bound what you missed.
The same JSON Schema can be offered to two providers and still produce different calls, costs and refusals. Keep one strict contract, write thin adapters, and run a paired experiment before you call the tool portable.
A fallback model runs rarely, so production traffic barely measures it and the aggregate hides it. Derive why, then force failover on your eval set and grade the fallback against the primary's contract.
With r retries at each of d layers, one request can become (r + 1)^d sends, and the multiplier peaks exactly when the dependency is down. Derive it, simulate it, and see what a shared breaker changes.
A counter in one process cannot stop eight workers from spending the same last dollar. Bound each call's maximum charge, reserve it atomically in a shared ledger, settle from real usage, and test the race.
Reverting the prompt alone can leave an old prompt talking to a new tool. Treat model, prompt, tools and config as one release tuple, roll back to a tuple you evaluated, and keep irreversible effects in their own ledger.
Learn
Save your progress with a free account.
Checking sign-in…
Save your progress with a free account.
Pick up where you left off.
Continue learning at my deskCookie duty
The one he did not eat is a measurement cookie. Say yes and the site loads Google Analytics (GA4), which stores an identifier in your browser and lets us count which lessons get read.
Change your mind any time with Privacy choices in the footer. Read the Privacy Policy
Optional · newsletter
That is saved and settled. Separately, and this is marketing rather than measurement: Rod writes a weekly email — one lesson, one field note — sent through Kit. It has nothing to do with cookies, and skipping it costs you nothing.
We will send a confirmation email. Click the link in it and the next lesson finds you.
