Forward KL Covers, Reverse KL Commits: Which Direction You Distill In
Setting: distillation trains a small student model to imitate a large teacher. The loss that compares them is usually KL divergence, a measure of how far one distribution is from another. It has a direction, and the direction changes what the student gets wrong.
Write p for the teacher and q for the student. Forward KL, the classic distillation loss, averages over what the teacher does: wherever the teacher puts probability, the student has to as well. Reverse KL averages over what the student does, so it punishes the student for putting probability where the teacher has little.
The MiniLLM paper (Gu, Dong, Wei and Huang, ICLR 2024) makes this the center of its method. Its Figure 2 fits a Gaussian mixture, the teacher, with a single Gaussian, the student, under each direction. It reports that minimizing reverse KL causes mode-seeking behavior, where the student assigns high probability to the teacher's large modes and ignores the small ones, while minimizing forward KL places large probability mass on regions where the teacher's probability is near zero. In text generation, the paper argues, that corresponds to generating low-quality text, so it trains the student with reverse KL, using an on-policy policy-gradient method.
A toy I computed on November 4, 2026 (arithmetic on a made-up teacher, not a model): the teacher has two equal bell-shaped peaks, centered at 12 and 29, each with standard deviation 2.5, on the points 1 to 40 (the curve is cut off at those points and rescaled to sum to 1). The student can only be one bell-shaped peak with its own center and standard deviation, cut off and rescaled the same way; I searched centers and widths on a grid. The best forward-KL student is centered at 20.5 with a standard deviation of about 10 (10 on my grid; a finer search gives about 10.2), mostly in the valley between the peaks. The best reverse-KL student sits on one peak, centered at 12 with standard deviation 2.5.
Pick the direction by asking which mistake costs more: missing something the teacher does, or inventing something it never does.
Source: Gu et al., "MiniLLM: Knowledge Distillation of Large Language Models", ICLR 2024 (the arXiv text I read is titled "MiniLLM: On-Policy Distillation of Large Language Models"), read November 4, 2026.
