A Dark Factory's First Control: Every Claim Carries Its Own Check

The factory's first control

A dark software factory is only as good as its controls, because no person reads each change on the way out. profrod.ai is built by one: zeo, a factory that applies the pattern beyond software, to the site's records, reviews and releases as well as its code. One operator directs several agent streams, and each stream works under a written statement of work (SOW): a markdown file that says what the work is, where it stands, and how anyone would know it is finished. The stream that wrote this revision has filed nearly seventy of them.

The first thing such a factory has to control is the oldest failure of coding agents: they are fluent at saying they are done. Devin, Factory's Droids and every agent in this site's factory will report success. The control is to make every claim of progress carry the check that would confirm or refute it, and to refuse the claims that do not.

Why a claim needs its own check

Suppose each handoff in a chain of agents passes along a claim that is right with probability p, and the claims fail independently. The chance that a chain of n handoffs is right all the way through is

P(all right) = p^n
0.95^10 = 0.599

Ten handoffs that are each right 95% of the time are all right about 60% of the time. The book's delegation chapter measures the same compounding across subagents. You cannot fix this by making the final reviewer more careful, because by the time the chain reaches the reviewer the evidence of which step went wrong is gone. You fix it by checking each claim where it is made.

That is the idea behind one line in every SOW on this site:

done_when: "<a command or observable condition> -> <the result that means finished>"

Two real stopping conditions from this site's records

Here is the done_when from the SOW that chartered this site's article publication gate, the rule that keeps a scheduled article hidden until its date:

A controllable-clock Site test proves that an article with a future Europe/London publishedAt is denied at its canonical route and absent from every enumerated discovery surface, then becomes available at its publish instant; invalid or ambiguous publication data fail closed; make verify exits 0.

A machine can check that one. It names a test, what the test must show, and an exit code.

Here, paraphrased, is the done_when on the reconnaissance SOW that started this rewrite: the operator has a verdict for every article, course, note and route, grounded in the files themselves, and an ordered plan. A person checks that one. It is still a stopping condition, because it says what must exist, but nobody can run it.

Both are honest. The difference matters: the first can be re-checked by anyone at any time; the second depends on the judgment of the person reading it. A research team faces the same choice when it writes down what "the experiment is done" means.

What the tool checks, observed on September 30, 2026

The records on this site are checked by a command-line tool called zeo, published on PyPI as zero-employee. Check PyPI for the version you will get. Everything below was run on this site's build machine, where zeo --version reports zeo 0.13.0, in a throwaway folder.

uv tool install zero-employee

The tool sorts a SOW's status into working statuses, which are owed attention (DRAFT, DESIGN, PROGRESS and one more), and resting ones (HELD, SHIPPED, FINDING and others). The rule that does the controlling is about the first family: a working SOW must carry a done_when and an estimate of the effort that remains.

So the refusal lives in doctor, and the plain lint only warns. If you wire this into a gate, gate on doctor.

View full-size diagram

The board is a view, not a record

The same records answer the operator's daily question, "what is in flight?" A board command walks every SOW, keeps the latest revision of each stream and writes a summary file. The summary is rebuilt on every run and never edited by hand. If the summary and a SOW disagree, the SOW is right, because the summary is derived from it.

That design choice carries over directly to agent systems. An agent's own summary of what it did is a view. The record is the files, commits and logs it produced. When they disagree, trust the record, and build the summary from it rather than asking the agent to remember.

Why this matters in a lab

Every company building a software factory faces this control first: Cognition, Factory and the coding-agent teams at every frontier lab ship agents whose reports of success have to be checked against something they cannot edit. Put two agents in a research repository and give them a week. One runs a sweep; the other writes up the results. Without a rule like done_when, "the sweep is done" means the agent stopped talking about it. With the rule, it means a named command produced a named result: a results file with one row per configuration, a seed recorded for each, and the evaluation run on the held-out split.

An interviewer for a research-engineering or agents role could ask: you are running work through coding agents; how do you know a result they report is real? A strong answer names the stopping condition before the work starts, checks each claim where it is made rather than at the end, and keeps the record separate from the agent's summary of it. The research engineering course is built around exactly those habits.

What would prove this wrong

If agent claims of completion turned out to be reliable enough that per-claim checks cost more than the errors they catch, this discipline would be overhead. The arithmetic above says that is unlikely for long chains, but it is an empirical question for short ones: at two handoffs and p = 0.99, the chain is right 98% of the time, and a lighter touch may be rational. Measure your own p before deciding.