Worked examples and arguments about AI systems: how they work, implementation choices, evaluation, costs, failures and what they change about organizations.
Build a workflow around your coding agent. The dark software factory workshop connects the explanations to one runnable application, acceptance checks and a release decision.
Archive
70 articles, newest first.
25–36 of 61 articles in Harness Engineering
Clear filters
Measured on 100 AIME problems, ten runs each: single runs of GPT-6 Luna scored anywhere from 88% to 98%. Derive task-level errors, pass@k and pass^k, and paired intervals, then play with them.
A repair loop needs three separate limits: attempts, a deadline, and external work. Derive what each extra repair buys, see why decode time sets the real cap, and measure repair success with an error bar.
An answer cache keyed on the query serves a revoked reader their old answer. Add an authorization revision to the key, recheck before release, test the race, and count what per-reader keys cost in tokens.
Measured on four small open models you can run on Colab: attack success from 10% to 61%, and a permission gate that let none through. Score it in code and bound it with the right sampling unit.
Send a model the field and error code it needs to repair a type error, stop on policy errors, and measure how much feedback detail actually improves repair success.
Coding agents sometimes pass by editing the tests instead of the code. Here is the evidence, the arithmetic, and a protocol you can run to measure how often it happens.
An injected line in a repo tells your eval agent the sweep is pre-approved for $5,000. Close the plan schema so no model output can set a budget, and test four override shapes.
A coding agent can implement a change to your eval runner. It cannot decide what counts as the same request, what a retry may do, or who may change the tests.
A notebook cell that parses one judge verdict is not yet a function a harness can call. Declare accepted inputs, rejected inputs and a stable output, then run eight cases through every entry point.
A research agent's validated sweep is a proposal, not a submitted job. Build the boundary between them, bind approval to exact code and config, and measure what it catches.
A signed, unexpired, correctly scoped token can still be meant for another server. Check audience, expiry and scope as separate predicates before the handler runs, and test each one with a positive control.
An eval or RL task is an input, an environment and a hidden grader. Freeze the environment, keep the oracle away from the agent, and prove the grader with null, cheating and gold agents.
Learn
Save your progress with a free account.
Checking sign-in…
Save your progress with a free account.
Pick up where you left off.
Continue learning at my deskCookie duty
The one he did not eat is a measurement cookie. Say yes and the site loads Google Analytics (GA4), which stores an identifier in your browser and lets us count which lessons get read.
Change your mind any time with Privacy choices in the footer. Read the Privacy Policy
Optional · newsletter
That is saved and settled. Separately, and this is marketing rather than measurement: Rod writes a weekly email — one lesson, one field note — sent through Kit. It has nothing to do with cookies, and skipping it costs you nothing.
We will send a confirmation email. Click the link in it and the next lesson finds you.
