With Only a Handful of Runs, Report an Interval and Use the Interquartile Mean
Setting: in reinforcement learning, a run is one training attempt of an algorithm from scratch, and results vary from run to run. Papers compare algorithms on suites of tasks, such as games, and usually report one aggregate number, like a mean or median score across tasks.
Agarwal, Schwarzer, Castro, Courville and Bellemare (NeurIPS 2021, "Deep Reinforcement Learning at the Edge of the Statistical Precipice") point out that this number comes from a finite, often tiny, number of runs. It is a random variable and should carry an interval. On the Atari 100k benchmark (26 games), they note that prior results were computed mostly from 3 or 5 runs, and they re-ran each of five algorithms 100 times to show how much point estimates move.
What they recommend: report interval estimates using a stratified bootstrap, which resamples runs within each task. And replace the mean, which a few outlier tasks can dominate, and the median, which has high variability, with the interquartile mean (IQM), also called the 25% trimmed mean: drop the lowest and highest quarter of the scores and average the rest.
A tiny example (arithmetic, not their data): eight scores, 0, 0.1, 0.2, 0.2, 0.4, 0.6, 0.7 and 12. The mean is 1.775, driven by the 12. The median is 0.30. The IQM, the average of the middle four, is 0.35.
They also release an open-source library, rliable, for these metrics.
The habit: next time you see "A beats B" from 3 or 5 runs, ask for the interval.
Source: Agarwal et al., NeurIPS 2021, read November 4, 2026.
