Record the Model Snapshot in Every Row of Your Results Table
A saved prompt is not an experiment record. The model snapshot, the sampling settings and the eval set can each move the score, and a results table that records only the prompt cannot tell you which one did. In a lab, "which checkpoint and which eval set?" is the first question anyone asks about a number.
OpenAI's prompt engineering guide says different snapshots within the same model family can produce different results. It recommends pinning production applications to a specific snapshot and building evals that measure prompt behavior when you change or upgrade model versions.
For every row of your results table, record:
- the model identifier the API returned in the response, not only the alias you requested;
- a hash of the prompt file and a hash of the eval set (
shasum -a 256 prompt.txt evalset.jsonl); - the sampling settings you sent, n and the date;
- each case's output and grade, not only the mean.
Change one thing at a time. If you swap the prompt and the snapshot in the same run, the improvement belongs to neither. When the snapshot is the one thing you change, count regressions and fixes item by item; this note is about the record that makes that comparison possible. Evaluation as measurement derives the paired comparison.
The check. Run the same record twice, unchanged. The two accuracies should agree within sampling noise. For n = 200 cases at p = 0.8, one run's 95% interval half-width is 1.96 × √(0.8 × 0.2 / 200) = 1.96 × 0.028 ≈ 0.055. The difference between two independent runs has √2 times the standard error, so its half-width is 1.96 × 0.040 ≈ 0.078. A gap well beyond that between identical records means something you did not write down is varying. Two runs over the same cases are paired, so compare them case by case as well; the book's evaluation chapter derives the paired comparison. The course repo has a misleading model comparison exercise to practice on.
