LoRA: Train Two Thin Matrices Instead of One Giant One

Setting: fine-tuning adapts a pretrained model to a task by updating its weights. For a very large model, updating all of them means storing a full copy per task, which the LoRA paper (Hu et al., ICLR 2022) calls prohibitively expensive. Its example is GPT-3 with 175 billion parameters.

LoRA, low-rank adaptation, freezes the pretrained weights and injects small trainable matrices into each layer. The idea for one weight matrix W of size d by k: instead of learning an update the same size as W, learn it as a product of two thin matrices, B (d by r) and A (r by k), where the rank r is small.

Count the numbers (arithmetic, not a training run). A 4096 by 4096 weight has 16,777,216 entries. With r = 8, B and A together have 8 × (4096 + 4096) = 65,536 trainable entries, which is 0.39% of the layer, or 256 times fewer.

The paper reports that against GPT-3 175B fine-tuned with Adam, LoRA can cut the trainable parameters by 10,000 times and the GPU memory requirement by 3 times, with quality on par or better on the models it tested and, unlike adapters, no extra inference latency.

A caveat for readers: those figures are the paper's own, on its chosen models and tasks. How much rank you need depends on the task, which is what the paper's rank-deficiency study investigates.

Source: Hu et al., ICLR 2022, abstract and introduction, read November 4, 2026.