For a Fixed Compute Budget, a Smaller Model Trained on More Tokens Can Win

Setting: training a large language model costs a fixed amount of computing, and you choose how to split it between a bigger model (more parameters) and more training text (more tokens). Get the split wrong and you waste part of the budget.

The Chinchilla paper (Hoffmann et al., NeurIPS 2022, "An empirical analysis of compute-optimal large language model training", the Chinchilla paper) trained over 400 models, from 70 million to over 16 billion parameters, on 5 to 500 billion tokens. It concluded that for compute-optimal training the model size and the number of training tokens should be scaled equally: for every doubling of model size, the tokens should also double. Models of the time, it argued, were significantly undertrained, and most of the largest had been trained on roughly 300 billion tokens.

The test: for the compute budget used to train Gopher (280 billion parameters), the paper predicted that a model 4 times smaller, trained on 4 times more tokens, would do better. It trained Chinchilla, 70 billion parameters on 1.4 trillion tokens, and reports that it uniformly and significantly outperformed Gopher, GPT-3, Jurassic-1 and Megatron-Turing NLG on a large range of downstream tasks.

Arithmetic of my own: 1.4 trillion tokens over 70 billion parameters is 20 tokens per parameter. A model of 280 billion parameters on about 300 billion tokens is roughly 1.

The paper adds that a smaller, better-trained model reduces inference cost, and that a language model's energy cost is amortized through its use for inference and fine-tuning.

A caveat: "compute-optimal" means best for training compute. If you will serve the model to many users, the best choice can differ.

Source: Hoffmann et al., NeurIPS 2022, abstract and introduction, read November 4, 2026.