Put the Rubric First, Then Measure Your Prompt-Cache Hits

Grading 2,000 eval items with the same rubric and the same few-shot examples means sending the same long prefix 2,000 times. That is the case prompt caching is built for. Whether your harness gets it depends on the order of your prompt, and whether it worked is a measurement, not an assumption.

OpenAI's prompt caching guide says caching is on by default for supported models and that reuse requires the entire rendered prefix to match. It recommends putting stable instructions and shared reference material first, and dynamic content such as timestamps or user-specific text at the end. For a grader, that means rubric, then examples, then the item under test. A run ID or date at the top makes every prefix unique. The prefix also has to reach a minimum length: 1,024 tokens for GPT-5.6 and later, and a value that varies by request settings for earlier models. A short rubric may never cache at all. Each response reports usage.input_tokens_details.cached_tokens.

The arithmetic. Let the prefix be P tokens, the item I tokens, and cached reads cost r times the uncached input price. Input cost per request, in uncached-token equivalents, is I + P·r on a hit and I + P on a miss. With P = 4,000, I = 500 and r = 0.1, that is 900 against 4,500. The guide lists r = 0.1 for most of the models it covers, and for GPT-5.6 and later it prices a cache write at 1.25 times the uncached rate. Across ten requests that share a prefix, one write plus nine reads costs 1.25 + 9 × 0.1 = 2.15 times the prefix's input cost, against 10 without caching. Look up both rates for your model. All of it holds only if the reads hit.

The check.

1. Send 30 items with a fixed 4,000-token prefix, rubric first and item last. Record cached_tokens / input_tokens and wall-clock latency for each response.
2. Repeat with one change: a run ID at the top of the prompt. Nothing else moves.
3. Report the fraction of requests with any cache hit in each condition, with an interval. For a fraction p at n = 30, the 95% interval is about 1.96 × √(p(1 − p) / 30), so ±0.18 at p = 0.6.
4. Record the model, the prompt version and the date with the result. A cache observation belongs to that configuration and does not carry over to another workload.

Expect the first request in each condition to miss, since nothing is cached yet. If rubric-first does not show the higher rate, check the prefix length and look for anything that varies near the top. Nobody has run this protocol for this note, so there is no result here to quote; the numbers above are arithmetic.

Two limits on chasing the rate. Do not move sensitive material into a shared prefix to raise it, and do not change the grading task to make it cache; compare cost per graded item at equal grading quality. If caching works and the bill barely moves, check what dominates it: long outputs, or retries that resend everything. Chapter 19 of the book derives cost per request.

OpenAI documentation.