Join Batch Eval Results by custom_id, Never by Line Number
Scoring 5,000 eval items through OpenAI's Batch API costs less than sending them one at a time. The guide lists a 50% discount against the synchronous APIs, with a 24-hour completion window. It also says the output line order may not match the input order, and tells you to use custom_id to map each result back to its request.
Join by line number and the failure is quiet. One missing or reordered line shifts every later answer onto the wrong item. The accuracy still comes out as a plausible number, and it describes nothing.
Give each request a custom_id equal to the item's dataset ID. When the batch finishes, reconcile three sets:
sent: the IDs in your input file;ok: the IDs in the output file;err: the IDs in the error file, found through the batch'serror_file_id.
Every sent ID should appear in exactly one of ok and err. None should appear twice, and nothing should appear that you didn't send. In set notation: ok ∩ err = ∅, ok ∪ err = sent, and no duplicates within ok.
The check. Test the reconciliation before the real run with a batch of 20 synthetic items, one of them deliberately invalid. Expected: 19 in ok, 1 in err, 0 missing, 0 duplicated. If the invalid request is handled some other way, write down how; learning that before the 5,000-item run is the point of the test. Save the reconciliation report with the results.
Plan for expiry too. The guide says that when a batch expires, unfinished requests are canceled and completed ones are still returned in the output file. Then ok ∪ err is smaller than sent, and your report has to say how many items were actually scored. An accuracy on 4,600 of 5,000 items is a different measurement, with a different n and possibly a different mix of items, so rerun the missing IDs rather than reporting the partial number as the whole.
Whether the batch path fits your deadline at all is a separate decision; this note assumes it does and is about scoring the right items. Exactly-once processing of work that can be lost or duplicated is the subject of Chapter 12 of the book.
