Why Attention Divides by the Square Root of the Key Dimension

Setting: in a transformer, attention lets each token weigh every other token. For each token it computes a query vector q, compares it with a key vector k for every token using a dot product, and runs the scores through a softmax to get weights that sum to 1.

Vaswani et al. (NIPS 2017, "Attention Is All You Need"; the conference is now called NeurIPS) call the version they use scaled dot-product attention: Attention(Q, K, V) = softmax(QKᵀ / √d_k) V, where d_k is the dimension of the keys. Why divide by the square root of d_k? The paper's footnote gives the reason. Suppose the components of q and k are independent random variables with mean 0 and variance 1. Then their dot product has mean 0 and variance d_k.

So with a larger key dimension the scores spread out. The authors suspect that for large d_k the dot products grow large in magnitude, pushing the softmax into regions where it has extremely small gradients, and they cite earlier work finding that additive attention beat unscaled dot-product attention at larger d_k. The scaling counteracts this.

A quick check (my simulation on November 4, 2026, with random standard normal vectors): the standard deviation of q·k was 2.0 at d_k = 4, 8.0 at 64 and 22.7 at 512, which is √d_k each time. After dividing by √d_k it was 1.0 in all three cases.

The habit: when a formula has an odd constant in it, ask what quantity it is holding fixed.

Source: Vaswani et al., NIPS 2017, section 3.2.1, read November 4, 2026.