Review a Model Upgrade: Count the Regressions
A new model snapshot can raise your eval score and still break items that used to pass. OpenAI's prompt engineering guide says it directly: "Even different snapshots of models within the same family could produce different results." Its advice is to pin production applications to specific snapshots and to build evaluation suites that measure prompt behavior "when you change and upgrade model versions."
Treat the upgrade as a diff. Run the pinned current snapshot and the candidate on the same frozen eval set, with the same prompt, and compare item by item. The headline difference hides two counts that matter more:
- b: items the current model got right and the candidate got wrong (regressions);
- c: items the current model got wrong and the candidate got right (fixes).
Suppose n = 300, b = 12 and c = 30. The accuracy gain is (30 − 12) / 300 = 6 points. Because both models answered the same items, the standard error of that difference is √(b + c − (c − b)²/n) / n = √(42 − 1.08) / 300 ≈ 0.021. The gain is 6 ± 4 points at 95%. Evaluation as measurement derives this paired interval and shows why it is tighter than comparing two separate accuracies.
The 12 regressions are the review. Read each one. If they cluster in one category, such as a tool the candidate stops calling or an output format it changes, that cluster is the finding, whatever the average says.
The check. Before your next upgrade, record b and c, not just two accuracies. File the list of regressed item ids with both model identifiers and the prompt version, in the record every results row should carry. Run the unchanged record twice first: if two identical runs already disagree beyond sampling noise, something unrecorded is varying, and the upgrade comparison has to wait. If the new model also needs a new prompt, evaluate that pair as its own configuration, so you know which change moved which items.
The numbers above are a worked example, not a measurement of any model.
