SFT Improved This Model. Did DPO Add Anything?

Lucy has fine-tuned a model to work arithmetic problems. Its accuracy is around 80%. A second training stage, direct preference optimization, optimizes a reference-relative preference loss and nudges recorded accuracy upward. Can she report an improvement?

That question has two parts. First, what did the new objective reward? Second, how much evidence supports the accuracy difference? We implement both stages on a small base language model, follow the tokens that receive gradients, and inspect Rod's recorded T4 results. SFT produces a large measured improvement over the three-example base model. DPO's incremental gain remains unresolved.

Entry skills: Python, token probabilities and basic gradients. The shifted next-token targets are explained in the SFT section below. Output: a CPU experiment that changes the reply boundary and preference margin, followed by a short write-up separating the objective, heldout accuracy and uncertainty. You do not need a GPU for that route.

The guided notebook starts with saved measurements and a standard-library exercise. Its optional full route reproduces the original model training. The companion includes explicit PyTorch loops, original receipts and standalone figures.

Fix the task before choosing the objective

The prompt asks for one arithmetic operation per line, ending with Answer: <number>. For 45 + 87 * 9, the supplied solution is:

87 * 9 = 783
45 + 783 = 828
Answer: 828

The generator obeys multiplication precedence and uses three operands. It produces disjoint expression strings for 3,000 SFT problems, 1,000 preference-sampling problems, 300 development problems, 1,000 test problems and three demonstration examples. All splits come from fixed seeds; the receipt pins their hashes. Disjoint expression strings prevent this experiment's train/test duplication. They cannot establish that the pretrained model never encountered similar arithmetic.

The grader extracts the first integer after Answer: and compares it with the known answer. It does not validate every worked step or require an otherwise correct explanation. Even a reply with a wrong trace can receive credit for the final integer. Accuracy therefore measures this answer-extraction task, rather than verified reasoning.

We start from SmolLM2-360M, the base model, not its instruction-tuned variant. SFT uses full fine-tuning, one epoch, batch size 16 and learning rate $10^{-4}$. DPO starts from each resulting SFT checkpoint, uses two epochs, batch size 16, learning rate $3\times10^{-6}$ and $\beta=0.1$. The earlier local MPS development-split calibration chose these settings; the T4 run reused them. Test comparisons did not choose a new learning rate.

The two base-model conditions differ in both examples and format: three worked examples in plain text versus zero-shot ChatML. Their 27.5% and 0.0% scores help locate the starting behavior. They do not isolate a single effect of the chat template.

SFT: predict the reply while reading the prompt

Let $x$ be a tokenized prompt and $y=(y_1,\ldots,y_m)$ its supplied reply, including the closing turn token. The model reads the prompt and earlier reply tokens. Supervised fine-tuning minimizes their negative log-probabilities:

Here $z$ concatenates prompt and reply, $\mathcal R$ contains reply target positions across the batch, and $N$ is their count. This implementation averages over scored tokens, rather than giving every differently sized example equal weight. Prompt tokens remain inputs; their labels are ignored as targets. They can still affect the reply predictions and the gradients flowing through the network.

The boundary is easy to misread after the causal shift. Use synthetic IDs to see it without downloading a model:

prompt = [10, 11, 12]  # includes assistant opening
reply = [20, 21, 2]    # ends with closing turn token
ids = prompt + reply
labels = [-100] * len(prompt) + reply
print(ids[:-1])     # [10, 11, 12, 20, 21]
print(labels[1:])   # [-100, -100, 20, 21, 2]

The last prompt input, 12, predicts the first reply token, 20. Masking that target would lose the first reply prediction. Scoring the first two targets would instead train on part of the prompt. Right-padding labels with -100 keeps padding out of the loss.

The actual loop makes the shift explicit:

loss = torch.nn.functional.cross_entropy(
    logits[:, :-1].float().reshape(-1, logits.size(-1)),
    labels[:, 1:].reshape(-1),
    ignore_index=-100,
)

SFT encourages the supplied continuation token by token. It neither compares that continuation with a rejected alternative nor directly optimizes the final-answer grader.

DPO: change a relative preference

For each of 1,000 fresh preference-sampling problems, each SFT model samples four replies at temperature 1. If a problem has both correct and incorrect replies under the answer grader, the code keeps one of each. Problems with only correct or only incorrect samples contribute no pair. Thus 1,000 problems does not imply 1,000 training pairs; the aggregate T4 receipt does not record the retained count.

Call the retained replies $y_c$ and $y_r$, for chosen and rejected. Their sampled token IDs are preserved, including any generated closing token. Re-tokenizing decoded text could change the sequence being scored. The implementation sums reply-token log-probabilities:

This is a sum, not a per-token average; changing that operation changes the objective and its sensitivity to reply length. Prompt and padding targets are excluded with a shifted scoring mask, just as in SFT.

The reference $\pi_{\mathrm{ref}}$ is that seed's SFT checkpoint. Before DPO updates, the code scores every fixed chosen/rejected pair under the SFT model, saves those reference values and holds them constant. This avoids keeping a second model in memory. It works because these training pairs remain fixed.

Define the relative margin

The DPO paper (Rafailov et al., NeurIPS 2023) gives the reference-relative preference classification objective. For our fixed pair, the implemented loss is

At initialization, policy and reference are the same: $m_\theta=0$ and the loss is $\log 2\approx0.6931$. Differentiating the scalar loss gives

The derivative is negative, so reducing the loss encourages a larger chosen-minus-rejected change relative to the reference. It does not require the chosen reply's absolute probability to rise. Neural parameters also couple many sequences; this scalar direction alone cannot predict heldout accuracy.

The loop uses stable log-sigmoid and summed reply log-probabilities:

margin = beta * ((pc - rc) - (pr - rr))
part = -torch.nn.functional.logsigmoid(margin).sum() / len(chunk)
part.backward()

pc and pr are policy scores; rc and rr are fixed reference scores. A 16-pair chunk is split into micro-batches of eight pairs. Each partial sum is divided by the whole chunk's pair count, gradients accumulate, and one clipped optimizer update follows. Averaging each micro-batch equally would misweight a shorter final micro-batch.

A better margin can accompany worse likelihoods

Take synthetic reference scores $\ell_{\mathrm{ref}}(y_c)=-10$ and $\ell_{\mathrm{ref}}(y_r)=-12$. Suppose training lowers the chosen score by 1 nat and the rejected score by 3 nats. Then $m=(-1)-(-3)=2$, so $\beta m=0.2$ and the loss falls from 0.6931 to about 0.5981.

Yet the chosen reply's probability becomes $e^{-1}\approx0.368$ times its reference probability. The rejected probability falls further, to $e^{-3}\approx0.050$. The preference improved relatively while both absolute likelihoods fell. These are synthetic scores illustrating the loss, not observed T4 model outputs.

Predict first: lower both reply scores. Which one must fall further for the preference loss to decrease?

Synthetic pair: fixed reference log-probabilities −10 (chosen) and −12 (rejected), with β = 0.1. These are whole-reply scores in nats, not token averages or recorded T4 results.

Change the policy scores; keep the reference and β fixed

Chosen score change:

0.00 nats

Rejected score change:

0.00 nats

Try −1 and −3Reset

Margin 0.00 nats; βm = 0.00; loss 0.6931. Chosen probability is 1.000 times reference; rejected probability is 1.000 times reference.

View full-size figure

Solid sea-glass circle: chosen reply. Dashed brass diamond: rejected reply. Zero means no change from that reply’s reference score.

Absolute policy scores and probabilities; no two-reply renormalization

Reply	Log-probability	Probability
Chosen	-10.00	4.540e-5
Rejected	-12.00	6.144e-6

Other replies can receive the probability mass. These two replies do not exhaust the possible continuations. A lower pair loss does not establish better answer accuracy.

Worked counterexample, also available without JavaScript

Changes −1 and −3 give margin 2, loss 0.5981 instead of 0.6931, chosen probability ratio 0.368 and rejected ratio 0.050. Both probabilities fall; the rejected reply falls further.

Change the two policy scores while keeping the reference and $\beta$ fixed. The two replies are not an exhaustive distribution: probability mass can move to other replies. The relative margin and pair loss cannot tell you whether held-out answer accuracy improved.

Predict the DPO gain

Check the prediction

What the T4 run measured

Rod's October 3 receipt records three training seeds on the same 1,000 test problems. Evaluation uses greedy decoding, a 64-token maximum and the answer grader described above.

SFT exceeds the three-example base model by 52.93 percentage points, with a recorded paired 95% interval of 49.87 to 56.00. It also passes the experiment's additional per-seed interval rule. This supports improved answer accuracy under the defined task and protocol; it does not prove that every worked step became correct.

DPO's seed gains over SFT are +0.7, +0.1 and +0.6 percentage points. Their mean is +0.47, with a recorded paired 95% interval of −0.63 to +1.57. The interval includes zero, and the receipt reports real: false. The incremental gain is unresolved. All three positive point estimates do not establish improvement, and the interval does not establish equivalence or zero effect.

Each line pairs one training seed. The seed-to-seed spread is larger than the incremental changes within seeds.

View full-size figure

Sea-glass circles: SFT. Brass diamonds: DPO. These are point estimates on the same 1,000 items, without error bars.

Recorded accuracy by training seed

Seed	SFT	DPO
0	79.2%	79.9%
1	81.9%	82.0%
2	80.2%	80.8%

Figure 1. Each line connects an SFT checkpoint with its DPO continuation. Sea-glass circles are SFT; brass diamonds are DPO. These are point estimates on the same 1,000 items, not confidence intervals. The seed-to-seed level changes are visibly larger than the incremental change within each seed.

Which uncertainty does that interval describe?

Let $a_{si}$ and $b_{si}$ indicate whether SFT and DPO, respectively, answer item $i$ correctly after training seed $s$. The paired difference for an item is

The statistics code resamples the item index 10,000 times, retaining each item's paired, seed-averaged difference. It takes percentile endpoints from the resampled means. Pairing matters because both models see the same item; the difference depends on which items they disagree about, not just their separate total accuracies.

This interval addresses item-sampling uncertainty conditional on the three fixed trained seeds. The bootstrap does not draw new training seeds. The 3,000 seed-item evaluations are not 3,000 independent test problems. The per-seed table reports a separate source of variation; neither the pooled item interval nor three observed seeds precisely characterizes performance after a fresh training run.

The experiment's real flag requires the pooled interval to exclude zero and every individual seed's paired interval to exclude zero in the same direction. This is a stated agreement rule, not a calibrated guarantee about all future seeds. The full T4 receipt preserves its verdicts, but omits individual seed interval endpoints and raw correctness rows. We can verify the means and split hashes; we cannot recompute the T4 paired bootstrap from these aggregate scores.

Change the implementation contract and measure again

The original experiment includes three controls against the same-seed SFT model. They ask different questions: one changes training labels, and two change the evaluation prompt.

The prompt-loss control trains the same kind of model but scores prompt targets too. That changes both the loss's terms and its token-count denominator. It allocates optimization effort to predicting the prompt's tokens as well as the supplied reply. The negative pooled interval supports lower answer accuracy for this fixed-seed comparison. It does not by itself isolate which mechanism caused the reduction.

The receipt still says real: false for prompt loss: the stricter per-seed agreement rule failed. This is compatible with a pooled interval entirely below zero. The supplied T4 receipt does not identify the failing seed interval; assigning it to a particular seed would invent evidence.

The two template-control intervals include zero. Their bounds describe the observed uncertainty under these prompts, this model and this task. We did not specify an equivalence margin, so neither establishes that the mismatch has no effect. A narrow arithmetic prompt also cannot establish robustness to arbitrary changes in the chat format.

Recorded 95% paired item-bootstrap intervals, each relative to SFT. The three training seeds stay fixed.

View full-size figure

Recorded accuracy differences and paired item intervals in percentage points

Change	Mean	95% interval
DPO	+0.47	−0.63 to +1.57
Prompt loss	−2.67	−4.07 to −1.33
System prompt	−0.10	−0.70 to +0.50
Opening off	+0.33	−0.10 to +0.77

The DPO interval includes zero. The pooled prompt-loss interval is negative, while its additional per-seed agreement rule fails. Neither result establishes equivalence.

Figure 2. Dots are mean accuracy differences; horizontal lines are recorded paired 95% bootstrap intervals over test items. The dashed vertical line is zero. All intervals condition on the same three training seeds. Prompt loss's pooled interval is negative even though the additional per-seed agreement rule fails. The other three intervals include zero.

Run the short experiment and write the claim

Use the notebook's first cells, or run the companion locally with Python's standard library:

python sftdpo_mechanism_v1.py

The helper checks the original T4 receipt's SHA-256, regenerates all five split hashes from the task code, verifies the recorded means and prints the synthetic mask/margin examples beside the saved comparison. It downloads no model, calls no API and performs no training.

Optional full reproduction and provenance

The original T4 run used float32 weights, activations and AdamW state. Recorded SFT peaks were 7.62–7.63 GiB; the largest recorded DPO peak was 9.71 GiB. SFT training took about 3.23 minutes per seed and DPO's recorded training sections about 3.83–4.00 minutes per seed. The 23 recorded sections total 41.42 minutes, excluding setup and other notebook overhead. This measured session supports feasibility on that T4; it does not guarantee free GPU availability or the same timing elsewhere.

The earlier MPS pilot and development calibration remain separate evidence. They are useful for inspecting raw generations and implementation diagnostics, but their item-level outputs cannot fill gaps in the T4 receipt. No new GPU training is needed to understand or report the comparison above.

Lucy can now report a strong SFT result and an unresolved incremental DPO result without confusing either with the preference loss. For the next measurement habit, work through evaluation variation across repeated runs: the unit you resample must match the claim you want to make.