Ask Your LLM Grader for the Smallest Output That Still Carries Evidence
An LLM grader in an eval pipeline often writes a paragraph of rationale for every item. Across a few thousand items, that paragraph is most of your wait, and most of it goes unread.
OpenAI's latency guide says output tokens dominate. Cutting 50% of your output tokens may cut about 50% of your latency, while cutting 50% of your prompt may improve it only 1 to 5%. The reason is the split between prefill and decode. The model processes the prompt in parallel, but generates the output one token at a time:
latency ≈ time to first token + (output tokens × time per output token)
Going from 300 output tokens to 30 removes 270 decode steps per item. Multiply by the number of items and you have the time you get back. Chapter 9 derives this split and shows where the time goes.
Shrink the contract, not the evidence. Ask the grader for three things: a label from a fixed set, one line quoted from the response being graded, and an unsure value when that evidence is not there. The quoted line keeps the grade checkable by a person.
The protocol. This course has not run it; here is how you would.
- Take 100 items with human labels.
- Run the long grader and the short grader on the same items.
- Record agreement with the human label, output tokens and latency per item.
- Compare agreement as a paired test. Count the items where exactly one grader matched the human; if those split evenly, the two graders agree equally well.
- Put an interval on each agreement rate: 1.96 × √(p(1 − p)/100), which is about ±0.06 at p = 0.9.
If the short grader loses agreement, keep the longer output. Speed that costs grading accuracy is a bad trade in an eval.
