Two Models Differ by a Point: Which Significance Test Do You Run?
Setting: you evaluate two models on the same test set, and one scores a point higher. Is that a real difference, or luck in which examples ended up in the test set? A significance test estimates how often a gap that size would appear by chance if the two models were truly equal.
Which test? Dror, Baumer, Shlomov and Reichart (ACL 2018, "The Hitchhiker's Guide to Testing Statistical Significance in Natural Language Processing") give a short protocol. Ask whether the test statistic comes from a known distribution. If yes, use a parametric test such as a paired t-test, which has more power. If not, and the data set is small, use a sampling-based test: the paired bootstrap, which resamples test examples with replacement, or a randomization test. If the data is large and sampling is too slow, use a sampling-free non-parametric test.
How well did the field do? The authors surveyed the long papers of ACL 2017 and the 2017 volume of TACL. Of the 180 empirical ACL papers, 117 (65%) reported no significance test. Of the 63 that did, 21 did not name the test, and of the 42 that named it, 6 used the wrong one.
Two cautions from the same paper: the bootstrap is less effective for small test sets, and the right test depends on the metric, because accuracy, F-score and BLEU behave differently.
The habit: when you report "A beats B by 1.2 points," also report the test, why it fits the data, and the p-value or interval.
Source: Dror et al., ACL 2018, pages 1383 to 1392, read November 4, 2026.
