Freeze the Eval Before You Tune the Prompt, Then Hold Some Out
If you decide what "better" means after reading each new answer, prompt tuning will always look like progress. Freeze the evaluation first. OpenAI's evaluation guidance calls this eval-driven development, "Evaluate early and often," and names the habit it replaces: "vibe-based evals."
Freezing is not enough on its own. If you try twelve prompt variants on the same dev set and keep the best, the winner's score is biased upward, because part of its lead is luck on those particular items. That is the winner's curse, and Optimizing against an evaluation derives it and measures it on a real prompt search.
The protocol.
- Write the labeled set before the first revision. Include ambiguous inputs whose correct answer is to decline or ask, and mark which errors are costly.
- Split it once, at random, with a recorded seed: for example, 200 dev items and 200 held-out items. Do not read the held-out items.
- Tune on dev only. Log every variant you try with its dev score, not just the winner.
- Score the chosen prompt once on held-out.
- Report both numbers with intervals. At p = 0.8 and n = 200, the 95% interval is 1.96·√(0.16/200) ≈ ±5.5 points.
What to expect. Because you selected the variant with the best dev score, its held-out score should come in lower on average. The size of the drop tells you how much of the tuning was fitting noise. If the drop is larger than the interval, you tried too many variants for a set that size, and the next step is more items, not more prompts.
Keep the items where the chosen prompt did worse than the original on dev, and read them. An average gain can hide a regression in a small category, and that category may be the one that matters.
