A 1.3B Model Tuned on Human Feedback Was Preferred Over a 175B One That Was Not

Setting: a language model trained only to predict the next word is not trained to do what a user asks. Ouyang et al. (NeurIPS 2022, "Training language models to follow instructions with human feedback") start from GPT-3 and fine-tune it in two stages. First, supervised learning on demonstrations that labelers wrote of the desired behavior. Second, they collect labeler rankings of model outputs, train a reward model on them, and use reinforcement learning from human feedback (RLHF) to optimize against it. They call the results InstructGPT.

The headline: on their prompt distribution, outputs from the 1.3 billion parameter InstructGPT model are preferred to outputs from the 175 billion parameter GPT-3, despite having over 100 times fewer parameters. Same architecture, different training data. Outputs from the 175B InstructGPT are preferred to 175B GPT-3 outputs 85 ± 3% of the time.

Limits to keep in mind: the preference is that of their labelers, on prompts that came from labelers and from users of the OpenAI API, not a universal quality score.

The habit: when a result says a method beats a much larger model, check who judged and on which prompts.

Source: Ouyang et al., NeurIPS 2022, read November 4, 2026.