Why Serving a Language Model Runs Out of Memory Before It Runs Out of Compute

Setting: a language model writes one token at a time. To avoid recomputing everything it has already read, it stores two vectors, a key and a value, for every earlier token at every layer. That store is the KV cache, and when serving a model it is often what limits how many users you can handle at once, more than the arithmetic does.

The size is multiplication. The vLLM paper (Kwon et al., SOSP 2023) works one example. For the 13-billion-parameter OPT model, the KV cache of a single token is 2 (key and value) × 5120 (hidden size) × 40 (layers) × 2 bytes (FP16) = 800 KB. A request that reaches 2048 tokens can need as much as 1.6 GB.

A back-of-envelope of my own, ignoring everything a real system does: if 40 GB of a GPU were free for the cache, it would hold about 24 requests of that length, not hundreds.

That is why the paper argues that serving throughput is memory-bound, and why managing that memory well, rather than adding compute, is what its PagedAttention method targets.

The habit: before you quote tokens per second, ask what the per-request cache is and how many requests fit.

Source: Kwon et al., SOSP 2023, section 3, read November 4, 2026.