Use the Batch API When Your Eval Can Wait a Day
A nightly sweep over 10,000 eval items and the debugging session where you rerun five failures have different deadlines. Pick the API path by the deadline first, then by the price.
OpenAI's Batch API guide gives the terms: a 24-hour completion window, a 50% cost discount compared with the synchronous APIs, and a rate-limit pool separate from your regular one. The saving is easy to compute. With 10,000 items at 1,500 input tokens each, the sweep sends 15 million input tokens. On the synchronous path that input costs 15 times the per-million input price; on batch it costs 7.5 times. Output tokens follow the same arithmetic. Look up the price for your model on the current pricing page rather than trusting a figure in a note.
The deadline is the constraint. The window promises completion within a day; it says nothing about this afternoon's meeting. If a batch expires, the guide says completed responses are still returned, and billed, while unfinished requests are canceled. So the pipeline needs a plan for the items that did not come back in time.
The check. Write down two times before you choose: when you can submit, and when you need the scores.
- If the gap is under 24 hours, the batch path makes no promise that fits, whatever its usual turnaround. Use the synchronous path, or submit earlier.
- If it fits, record submission and completion times for every run, so the turnaround you plan around is one you measured rather than one you remember.
- Test the fallback before you need it. Remove five ids from a finished output file and confirm the pipeline marks the run incomplete and resubmits exactly those five on the synchronous path, before anyone computes an accuracy.
Getting the results back onto the right items is a separate source of quiet errors: the guide warns that output line order may not match input order, and failed requests go to their own error file. Join by custom_id and reconcile the ids before you score anything.
What a model call costs works through cost per accepted result, which is the number to compare across the two paths.
