In a Dark Factory, Checking Is the Real Price of an Answer

The question that has no single answer

The price of a model call keeps falling, and every frontier lab publishes its prices per million tokens. It is tempting to conclude that a dark software factory, where agents produce the work and checks decide what ships, gets cheap in proportion. This essay argues something narrower and more useful: as inference gets cheaper, the price of an output you can use is set more and more by the cost of checking it. That holds for code, and for every other kind of work a factory can be given once the work is written as code.

Take a routing decision. One model costs less per call; another needs fewer corrections. Which belongs in the workflow? The average bill per run is a real number, but it does not answer that on its own. Cost accounting has long separated several questions that one average blurs:

What did this absorb of total cost? is a different question from what does one more unit add?, which is different again from what would disappear if we stopped doing this altogether?

A number that is useful for allocating the total bill may be the wrong one for deciding whether to run one more task. Inference can be metered in tokens; review consumes a person's time. A useful record keeps both, along with whether the result was accepted. A token log alone is not that record.

The arithmetic of an accepted answer

Let each attempt cost c in inference and R in review (minutes of review times what a minute of that reviewer's time is worth). Let each attempt be accepted with probability p, independently, and retry until one is. The number of attempts is geometric with mean 1/p, so

cost per accepted answer = (c + R) / p

Now put in illustrative numbers, in USD. None of these are measurements; the value of a review minute in particular is an assumption you must state.

review: 4 minutes at an assumed $1.00 per minute  ->  R = $4.00

model A: c = $0.01, p = 0.80   ->  (0.01 + 4.00) / 0.80 = $5.01
model B: c = $0.03, p = 0.95   ->  (0.03 + 4.00) / 0.95 = $4.24

Model B costs three times as much per call and is about 15% cheaper per accepted answer. The token price is under 1% of either total. When c is small next to R, A's cost divided by B's approaches p_B / p_A (here 0.95 / 0.80 ≈ 1.19, against the exact 5.01 / 4.24 ≈ 1.18), and the acceptance rate decides everything.

For an operational comparison, count outputs accepted against the task's criteria, not outputs generated. State what the checks cover. Passing a schema check is not evidence that every factual claim is correct.

The error bar that decides the route

The acceptance rates above are estimates, and the decision depends on them. Suppose you reviewed 50 outputs from each model. A 95% interval for a proportion is about 1.96 × √(p(1 − p)/n):

model A: 1.96 × √(0.80 × 0.20 / 50) = 0.111   ->  0.69 to 0.91
model B: 1.96 × √(0.95 × 0.05 / 50) = 0.060   ->  0.89 to 1.00

The intervals overlap. Fifty reviews each cannot tell you that B is better, and so cannot tell you it is cheaper. This simple interval is also unreliable close to 1, which is where model B sits; Chapter 16 of the book measures how often a nominal 95% interval covers the truth and derives a better one. The stronger design is a paired comparison: run both models on the same tasks, so each task's difficulty cancels out of the difference.

What your cost-per-task dashboard leaves out

View full-size diagram

If your dashboard divides total spend by run count, inspect what that total includes: task calls, orchestration, retries and standing context may be mixed together. Using the resulting average for every task allocates those costs by volume.

Suppose one task needs a single call and another needs several calls plus a long review. A shared average hides that difference. Keep the observations separate before comparing routes. Standing costs change the picture too. If a fixed subscription is spread across fewer runs, its allocated cost per run rises even though the next run consumes the same resources. Cutting useful tasks in response raises that average again.

Tokens are the visible, metered part. Measure the other inputs before assuming they are smaller. Minimizing the metered factor because it is metered is looking for your keys under the streetlight. The companion essay on review queues applies the same arithmetic to a whole queue, where the acceptance rate is the variable that moves.

Cost attaches to a decision, not to a thing

When someone says "an agent costs X per task," ask which costs they included: marginal inference, subscriptions, orchestration or development. The same label can hide different accounting choices.

The missing discipline is attaching each cost to the level of decision that causes it:

A task can pay for itself while the stream it belongs to does not cover its own readiness costs. Mixing the levels misleads in both directions: an allocated charge can make a useful extra task look unattractive, while a low incremental cost can hide expensive standing capacity. A workflow can be cheap to run once and expensive to keep ready.

Isn't this just Taylorism on the factory floor?

The objection is serious, and the answer is not obvious.

Frederick Taylor timed factory workers with a stopwatch, and the history of that is not kind to the people being timed. A dark factory has fewer people on the floor, but the ones it has are the reviewers, and the same concern applies to how a measurement of them gets used. A review-time record can show that a workflow creates extra checking. The same record can become a way to rank or pressure the people doing the checking. This essay argues for diagnosing the workflow, with the reviewers involved in deciding what is measured and how the record may be used. Logs may also contain personal or confidential data from the task; an agent's involvement does not remove the need to limit access, retention and content.

What smart people get wrong

What to actually track

Four fields per work product. A short manual record can be enough:

Why this matters in a lab

Labs price their models per token, and their customers turn that into a cost per task. An applied-AI engineer helping a customer choose a model is doing the calculation above, often without the review-time term. Leaving it out is the most common way a cheaper model gets chosen and turns out more expensive.

An interviewer could fairly ask: a customer wants to switch to a model that is three times cheaper per token. What do you need to know? A good answer writes down (c + R) / p, asks for the review time and acceptance rate on the customer's own tasks, and says how many paired examples it would take to tell the two rates apart.

What would prove this wrong

If review cost fell toward zero as fast as inference cost, for example because automated checks with measured, high agreement replaced most human review, the token price would matter again and this essay's emphasis would be misplaced. That is worth watching: a cheaper check lowers R for every model at once.