Optimize Against a Reward Model Too Hard and the True Reward Falls

Setting: to train a language model on what people prefer, you fit a reward model, a second model that predicts which of two answers a person would pick, and then optimize the language model to score well on it. The reward model is a proxy for human judgment. Goodhart's law says what happens when you push a proxy too hard: "When a measure becomes a target, it ceases to be a good measure."

Gao, Schulman and Hilton (ICML 2023, "Scaling Laws for Reward Model Overoptimization") measured this. Because collecting human labels is expensive, they used a synthetic setup: a fixed "gold" reward model plays the role of humans and labels the data used to train smaller proxy reward models. They then optimized against the proxy, with reinforcement learning or with best-of-n sampling (draw n answers and keep the proxy's favorite), and tracked the gold score.

The gold score rises, then falls as optimization continues. For best-of-n they report a functional form R(d) = d(α − βd), where d is the square root of the KL divergence between the optimized and the original model, and the coefficients scale smoothly with the reward model's size.

Toy arithmetic with invented coefficients α = 1 and β = 0.25 (not their fit): R is 0.75 at d = 1, peaks at 1.0 at d = 2, and is back to 0 at d = 4. Same optimizer, more optimization, no gain.

They also report that, as a function of KL, reinforcement learning is slower than best-of-n at both optimizing and overoptimizing.

The habit: judge a trained model on a measure the optimizer never saw, and stop where that measure peaks.

Source: Gao et al., ICML 2023, read November 4, 2026.