Latest

Thu, Oct 8, 2026Thursday, October 8, 20261 entry

EssayMeasure models· Food for Thought

A Job ID Is Not a Result: Check the Output Before Marking It Done

A scheduler's job ID, like a deploy hook's 200, says a request was accepted. A five-state model and a small check that marks work done only when the results file exists, from the right checkpoint, with every row.9 min readexperiment pipelinesevaluation infrastructure

Mon, Oct 5, 2026Monday, October 5, 20261 entry

EssayMeasure models· Food for Thought

A Subagent's Report Is Not a Handoff Until the Receiver Can Act

A subagent's report can arrive while everything it learned stays behind. Define handoff acceptance by what the receiver can check and do next, whether it is an orchestrator or a worker resuming a crashed eval run.9 min readhandoffsmulti-agent systems

Sat, Sep 12, 2026Saturday, September 12, 20261 entry

EssayBuild systems around models

Measure Whether a Coding Agent Weakens Tests Under Pressure

Coding agents sometimes pass by editing the tests instead of the code. Here is the evidence, the arithmetic, and a protocol you can run to measure how often it happens.10 min readspecification-gamingevaluation

Fri, Sep 11, 2026Friday, September 11, 20261 entry

EssayBuild systems around models

What You Still Decide When a Coding Agent Builds the Change

A coding agent can implement a change to your eval runner. It cannot decide what counts as the same request, what a retry may do, or who may change the tests.8 min readdark-software-factoriescoding-agents

Thu, Sep 10, 2026Thursday, September 10, 20262 entries

EssayMeasure models· Food for Thought

A Contamination Cutoff Needs a Clock, Not Just a Date

A training cutoff of March 31 is a date without a clock. One post can land on either side of it depending on the time zone. How to pick one clock, size the ambiguous boundary day and keep it out of a held-out split.8 min readevaluationtimezones

EssayMeasure models· Food for Thought

Catch Duplicate Eval Results Before a Dictionary Hides Them

A retry appends a second result for the same case. Build a dictionary keyed by case ID and the pass rate now depends on row order; keep both rows and the error bar shrinks for no reason. Check the list first.8 min readevaluationdata integrity

Mon, Aug 17, 2026Monday, August 17, 20261 entry

EssayBuild systems around models· Food for Thought

A Dark Factory's First Control: Every Claim Carries Its Own Check

Coding agents are fluent at saying they are done. In the dark factory that builds this site, every claim of progress must name the check that would end it, and a gate refuses the ones that do not. What that control catches, and what it cannot.6 min readdark-software-factoriesAI control

Fri, Aug 7, 2026Friday, August 7, 20261 entry

EssayBuild systems around models· Food for Thought

What a Dark Factory Must Build Itself When Agents Are Cheap

Every station in a dark software factory is a make-or-buy decision. Coase and Williamson worked out the answer decades before agents: buy what is generic, build what is specific to you, and keep the right to decide when the specification runs out.8 min readdark-software-factoriesmake or buy

Thu, Aug 6, 2026Thursday, August 6, 20264 entries

EssayMeasure models· Food for Thought

Clean an Eval Set With a Coding Agent, Then Prove It Dropped Nothing

An eval set merged from three sources has mixed label spellings, near-duplicate prompts and empty references. A coding agent will tidy it. Three counts, computed by your own script, show whether it dropped a row that would fake a gain.8 min readevaluationdata-quality

EssayBuild systems around models· Food for Thought

In a Dark Factory, Checking Is the Real Price of an Answer

As model calls get cheaper, what a dark software factory's output costs comes to be set by review time and acceptance rate, not tokens. The arithmetic, the error bar that decides a route, and the accounting that keeps it honest.8 min readeconomics of AIinference-cost

EssayBuild systems around models

Test What a Claude Code Permission Rule Actually Allows, in a Sandbox

A Claude Code permission rule is a string pattern, not an intent. Write the commands it should stop, run them through a headless session in a throwaway VM, and count the unintended allows before you trust the rule.11 min readclaude-codeagent-security

EssayBuild systems around models· Practical Guides

Using a Coding Agent on Research Code: Ask Where the Expected Values Came From

An agent can refactor an eval script and pass every test because it rewrote the expected values. A short guide to using Claude Code or any coding agent on research code and take-homes, built around that check.7 min readclaude-codecoding-agents

Wed, Aug 5, 2026Wednesday, August 5, 20261 entry

EssayBuild systems around models· Food for Thought

A Dark Factory Beyond Software, and the Review Arithmetic That Caps It

A dark software factory turns requests into tested software with no person on each change. The same pattern runs any work you can write as code. What caps it is review, not compute: the arithmetic, the control layers, and where it breaks.9 min readdark-software-factoriesverification

That is the first entry in the log.

To browse rather than follow: topics, all articles and all notes. To learn in order: courses and the book.