A Verifier That Checks Each Step Beats One That Checks Only the Final Answer
Setting: a language model can solve a math problem by writing its reasoning step by step, and you can raise its accuracy by sampling many solutions and letting a second model, a verifier or reward model, pick the best one. The question is how to train the verifier. Outcome supervision trains it only on whether each solution's final answer is right. Process supervision gives feedback on every step.
Lightman et al. (ICLR 2024, "Let's Verify Step by Step") compared them on MATH, a dataset of competition problems. They collected human labels for each step and released the dataset, PRM800K, with 800,000 step-level labels. They supervised only up to the first incorrect step, so both methods carry the same information for correct solutions, and the process labels add only the location of the mistake in wrong ones.
The result, from their Figure 3: with 1,860 sampled solutions per problem, the best-of-N accuracy on a representative subset of the MATH test set was 78.2% for the process-supervised reward model, 72.4% for the outcome-supervised one, and 69.6% for majority voting, a strong baseline that needs no verifier.
They also report that active learning, choosing which solutions to label, improved the data efficiency of process supervision by 2.6 times.
Caveats the paper states itself: the large-scale comparison is "not ideal for making an apples-to-apples comparison," because the outcome model trained on 100 uniform samples per problem, an order of magnitude more data than the process labels, and the two training sets are not directly comparable. A smaller-scale setup, supervised by a large model, gives a more direct comparison. On AP and AMC exam questions released after the model's pretraining data was compiled, the same ordering held (best-of-100: process 72.9%, outcome 63.8%, majority voting 61.3%).
The habit: when a pipeline samples many candidates and selects one, report how good the selector is and compare it against a no-verifier baseline such as majority voting.
Source: Lightman et al., ICLR 2024, read November 4, 2026.
