Worked examples and arguments about AI systems: how they work, implementation choices, evaluation, costs, failures and what they change about organizations.
Build a workflow around your coding agent. The dark software factory workshop connects the explanations to one runnable application, acceptance checks and a release decision.
Archive
70 articles, newest first.
13–24 of 70 articles
Newest first
A stored fact goes stale when time passes or when its source changes. Record who owns it and which revision it came from, select the current fact by rule rather than similarity, and escalate conflicts.
An exact, byte-matched quote proves where text came from, not that it supports the claim. Test provenance and entailment separately, and compare exact match, NLI and an LLM judge.
Measured on SciFact with small open models: attribute each miss to retrieval, reranking, prompt assembly or generation, then change only the stage that failed. A targeted prompt fix took 107 correct to 151.
Write boundary cases for an eval grader from its policy, compute expected results by hand, then break the grader on purpose to prove every row can fail. Seven mutants, seven kills.
A truncated agent transcript can look like a clean success. Record what a trial needs for grading and replay, mark truncation explicitly, and size how many transcripts to read.
Agent traces are the raw material for evals, and production traces carry user data. Capture structure by default, keep content in a restricted store, test every serialization path with canaries, and bound leaks by sampling.
A canceled eval run still has requests in flight, bills for them, and leaves a biased partial score. Name the states, count the in-flight work with Little's law, and report only what the evidence supports.
An approval is a check; the agent's action is the use. Close the time-of-check-to-time-of-use gap: bind approval to exact state, estimate how often it goes stale, and make the final check atomic.
On October 2, 2026, the jobs board counted 1,837 open technical roles at 49 labs. Here is what research and applied postings mention most, then fifteen postings read in the labs' own words.
After a restart, an agent remembers what your application stored and nothing else. Run a short Python example on Colab that loses its conversation and keeps a corrected preference.
An experiment-tracker timeout that returns [] tells your agent no prior runs exist. Keep failure and emptiness apart with typed results, a mutation test, and a protocol to measure the damage.
A tool whose name hides a destructive branch makes the dispatcher's job and the model's job harder. Split tools by effect class, then measure merged against split under injection, with intervals.
Learn
Save your progress with a free account.
Checking sign-in…
Save your progress with a free account.
Pick up where you left off.
Continue learning at my deskCookie duty
The one he did not eat is a measurement cookie. Say yes and the site loads Google Analytics (GA4), which stores an identifier in your browser and lets us count which lessons get read.
Change your mind any time with Privacy choices in the footer. Read the Privacy Policy
Optional · newsletter
That is saved and settled. Separately, and this is marketing rather than measurement: Rod writes a weekly email — one lesson, one field note — sent through Kit. It has nothing to do with cookies, and skipping it costs you nothing.
We will send a confirmation email. Click the link in it and the next lesson finds you.
