The DPO Loss Is One Number: How Much More You Raised the Winner Than the Loser

Setting: to make a language model prefer better answers, you collect pairs: a chosen answer and a rejected answer for the same prompt. The older recipe, RLHF, first fits a separate reward model and then runs reinforcement learning. Direct Preference Optimization (Rafailov et al., NeurIPS 2023) shows the same problem can be solved with one classification-style loss.

The loss in words: take how much the model being trained has raised the probability of the chosen answer, relative to a frozen reference model. That is a log ratio. Subtract the same quantity for the rejected answer, and multiply by β, a constant that sets how far the model may drift from the reference. Feed that margin through a sigmoid and take the negative log: loss = −log σ(β × margin).

Numbers (arithmetic, not a training run), with β = 0.1, one of the values the paper tests: a margin of 0 gives a loss of 0.693. A margin of +0.5 gives 0.668. A margin of −0.5, the model ranking the pair backwards, gives 0.718. With β = 1 the same margins would give 0.474 and 0.974.

The paper's gradient analysis adds a detail. Each pair is weighted by how wrongly the implicit reward orders it, so backwards pairs get larger updates. The authors report that a version without this weighting can make the language model degenerate (their Appendix Table 3).

The takeaway: when you read a DPO training curve, the loss starts near 0.693 because the margin starts at 0, and it only moves if the chosen answer gains on the rejected one relative to the reference.

Source: Rafailov et al., NeurIPS 2023, read November 4, 2026.