Language Models Memorize More as They Grow, as Text Repeats, and as the Prompt Lengthens

Setting: a language model is trained on a huge pile of text, and sometimes it can reproduce parts of that text word for word. That matters for privacy, because it can expose personal data, and for quality and fairness, because some texts get memorized and others do not.

Carlini et al. (ICLR 2023, "Quantifying Memorization Across Neural Language Models") measured it. They took sequences from the training set of the GPT-Neo models, fed the model all but the last 50 tokens, and called a sequence "extractable" if the model emitted those 50 tokens exactly. They found three log-linear relationships: memorization grows with the capacity of the model, with the number of times an example was duplicated in the training set, and with the number of tokens of context in the prompt.

The numbers they report. For model size, on their evaluation sample, a ten-fold increase in model size corresponds to an increase in memorization of 19 percentage points. They note that their sampling over-represents duplicated strings, so the trend matters more than the absolute level. For context, 33% of the training sequences in their evaluation set are extractable from the 6B model with 50 tokens of context, against 65% with 450 tokens. They call this the discoverability phenomenon: some memorization only shows up when the prompt is long enough.

They also report that the growth with size is not just better generalization, since a GPT-2 baseline does not show it.

The implication they draw: memorization is more prevalent than previously believed and will likely get worse as models scale, without active mitigations.

The habit: if you test whether a model leaks its training data, test with long prompts, or you will undercount.

Source: Carlini et al., ICLR 2023, read November 4, 2026.