Compare Cost per Accepted Result, Not per Token or per Session
A cheaper model, or a cheaper Claude Code setup, can cost more per result you keep. A call or a session whose output you throw away cost money and bought nothing. The number to compare is cost per accepted result, where "accepted" means it passed a check you wrote in advance.
The derivation. One attempt costs c; for an API call, c = (input tokens × input price) + (output tokens × output price). Suppose a fraction a of attempts pass. If you retry until one passes, the number of attempts is geometric with mean 1/a, so the expected cost per accepted result is c/a. If a person reviews every attempt, add review time × hourly rate to c before you divide.
With round numbers: an option whose attempts cost a quarter as much, but pass half as often, costs (1/4)/(1/2) = 1/2 as much per accepted result, not a quarter. At a tenth of the pass rate, it costs (1/4)/(1/10) = 2.5 times as much. Chapter 19 derives where c comes from, from prefill and decode to cost per task.
Reading c. For API calls, take input and output tokens from the usage fields. For Claude Code, the costs documentation says the Session block in /usage shows the session's token usage and a dollar figure computed locally from token counts at list price. It is an estimate, not a bill, and for Pro and Max subscribers the session cost figure is not relevant for billing. To compare two setups, an estimate at the same prices is enough. OpenAI's model-selection guide puts the method in one line: experiment using the same inputs to compare results, and keep the lightest setting that meets your quality bar. Its model list changes, so read it on the day you run the comparison.
The protocol. Run both options on the same items from the same starting point, with the acceptance check fixed before you start. For a grading or labeling pipeline, that might be 100 items and one fixed prompt. For two Claude Code setups (a different model, a different CLAUDE.md, plan mode on or off), 10 tasks from your repo's issue list, each started from the same commit. Record cost, accepted or not, and your cleanup minutes. Report c, a with its interval, and c/a for each option.
As an illustration of the arithmetic, not a measurement: setup A spends US$4.20 over 10 tasks with 6 accepted, US$0.70 per accepted change. Setup B spends US$3.00 with 3 accepted, US$1.00. The setup that was cheaper per session cost more per result.
Put intervals on a before you decide. For 6 of 10, 1.96 × √(0.6 × 0.4 / 10) ≈ ±0.30; for 3 of 10, ±0.28. The intervals overlap, so 10 tasks do not settle the question, and at n = 10 the formula itself is rough. At n = 100 and a = 0.8 the half-width is about ±0.08. Because both options ran the same items, the items where one passed and the other failed carry the evidence, and a paired comparison on them needs fewer items than two independent rates; Chapter 16 derives it. Split routine items from hard ones too: an option can win on the first and lose on the second, and the hard cases are often why the pipeline exists.
Nobody has run either protocol for this note.
