Test a Model on the Hard Slice, and Put an Interval on It
An eval average can look healthy while the model fails the cases that decide whether it is fit for the task. Suppose your set has 200 typical items the model gets 95% right and 40 hard items it gets 60% right. The headline is (190 + 24) / 240 ≈ 89%. The slice that matters is 60%, and the average buried it.
OpenAI's evaluation guidance asks you to make sure your "test data includes typical cases, edge cases, and adversarial cases." In applied-AI and research work, the next step is the question a careful reader of your results table asks first: how many items sit in each row, and how noisy is each number?
The protocol.
- Tag every item with a slice before you run anything: typical, edge or adversarial. Write the expected answer first.
- Run the model over the set and report accuracy per slice with its count n.
- Put a 95% interval on each slice: 1.96·√(p(1−p)/n). For the hard slice at p = 0.6 and n = 40, that is 1.96·√(0.24/40) = 1.96 × 0.077 ≈ 0.15, so 60% ± 15 points.
- When you compare two models, score both on the same items and count the items where they disagree. That paired comparison is tighter than two separate intervals. Evaluation as measurement derives it.
- Keep a held-out slice you do not look at while you tune.
The check. Compute step 3 for your smallest slice. If the interval is wider than the difference you hope to detect, add items to that slice before you claim anything about it. A model that is 10 points better on the hard slice cannot be told apart from one that is not, at n = 40.
This note reports no model scores. The numbers above are arithmetic on an example, there for you to check.
