One Number Can Fix an Overconfident Classifier: Temperature Scaling
Setting: a classifier outputs a probability for its answer. If it is calibrated, the answers it gives 90% confidence are right about 90% of the time. That matters whenever someone acts on the confidence, for example to decide when to ask a human.
Guo, Pleiss, Sun and Weinberger (ICML 2017, "On Calibration of Modern Neural Networks") found that modern neural networks, unlike those from a decade earlier, are poorly calibrated, and their example network is overconfident. They measure it with Expected Calibration Error (ECE): bucket predictions by confidence and average the gap between each bucket's confidence and its actual accuracy.
Their practical recipe is temperature scaling. Take the logits z, the raw scores before the softmax, divide them by a single number T, and apply the softmax. T is fit on a held-out validation set by minimizing negative log-likelihood. Because dividing by T does not change which class has the largest score, accuracy does not change. With T above 1 the output softens.
Toy arithmetic (mine): logits 2.0, 1.0 and 0.1 give probabilities 0.659, 0.242 and 0.099 at T = 1, and 0.502, 0.304 and 0.194 at T = 2. The ranking is the same, and the confidence is lower.
Their Table 1 reports, for a ResNet 110 on CIFAR-100, an ECE of 16.53% before calibration and 1.26% with temperature scaling, using 15 bins.
A caveat: the paper's results are for image and document classifiers, and a single T assumes the validation set looks like the data you will see.
The habit: if a system acts on a model's confidence, measure calibration on held-out data before you trust the number.
Source: Guo et al., ICML 2017, read November 4, 2026.
