Context Windows and the Shape of Forgetting

Class 2 told you the context window is finite and degrading, and asked you to take both properties on trust. This unit pays that debt. You'll see mechanically why a window degrades as it fills, not just that it does, and you'll meet a specific, measured shape that finite degradation takes: information buried in the middle of a long context is the least reliably recalled, regardless of how capable the model is. That shape has a name, it has a cause you can actually reason about, and it has direct consequences for how you'll structure everything Xolo's organization reads from here on.

Attention has a budget, and softmax is how it's spent

Every token a model generates draws on every other token currently in the context window through a mechanism called attention: for each token being generated, the model computes a relevance weight against every token already present, then blends information from all of them, weighted by relevance. The mechanism that turns raw relevance scores into weights that sum to one, so the model is dividing a fixed budget of attention rather than an unlimited one, is called softmax.

That "sum to one" detail is the whole story, so it's worth sitting with. If a context window holds twenty tokens, attention has twenty slices of a fixed pie to hand out. If it holds twenty thousand, attention still has exactly one pie, now sliced twenty thousand ways. Nothing about softmax changes as the window grows except how thin each individual slice gets. A fact stated once in a ten-thousand-token document doesn't get less true as the document grows, but the fraction of the model's attention budget available to notice it, on any given generation step, shrinks every time something else gets added to the window. This is the literal mechanical cause of "the context window is degrading" from Class 2, not a metaphor for it.

The U-shaped curve: lost in the middle

Attention dilution alone would predict output quality degrading smoothly as context grows. What researchers actually measured, in the paper that gave this effect its name, Liu et al.'s "Lost in the Middle," is stranger and more specific: recall accuracy across a long context doesn't degrade smoothly. It forms a U-shape. Information placed at the very start of a context and information placed at the very end are both recalled reliably, close to what a short context would achieve. Information placed in the middle is recalled measurably worse, sometimes dramatically so, even though nothing about the fact itself changed, only where it sat in the window.

The mechanism behind the two peaks has real names in the literature: primacy, an advantage for what appears early, echoing how attention patterns get established from the start of a sequence, and recency, an advantage for what appears last, closest to the point where generation is actually happening. Everything between those two peaks sits in the trough of the U, competing for a thinning attention budget without either advantage.

Picture this against Xolo's own reconciliation report generation. If the currency-mismatch deposit sits on row 4 of a 200-row transaction log pasted into context, or on row 197, it has a real structural advantage over the same exact row sitting at row 100. Nothing about the deposit itself changed. Only its position in the window did. This is not a hypothetical worry, it's the direct, measured reason document position is a design decision in this course, not an afterthought, and it's the entire premise of the lab in the next unit.

Context rot: the 2025 finding that makes this worse, not better

A reasonable hope, going into 2025, was that newer models with dramatically larger context windows would simply engineer this problem away, more room means less crowding, less crowding means less dilution. Research published in 2025 by Chroma Research, under the name context rot, found close to the opposite for a specific and important class of task. Treat the specific numbers in that report as measured for the models and tasks it tested, not as a universal constant for every model or every context length; what this course actually relies on is the direction of the effect, not any single figure. As tasks require reasoning across more of a long context rather than just retrieving one isolated fact from it, degradation doesn't merely persist at longer lengths, it compounds. A model asked to synthesize information scattered across many parts of a long document shows measurably worse reliability than the same model asked to retrieve one clearly-marked fact from a context of the same length. Bigger windows didn't dissolve the problem. They changed its shape: simple lookup got somewhat more forgiving over time, while actual multi-part reasoning across a long context stayed hard, and in some measured cases got relatively worse as length grew.

This matters directly for how Xolo's organization gets designed. A task like "does this one specific deposit exist in the bank file" is single-fact retrieval, and a large context window handles it with reasonable grace. A task like "reconcile every transaction across two hundred rows, cross-referencing invoice status, deposit currency, and contract terms simultaneously" is exactly the multi-part reasoning that context rot shows degrading hardest. That's the entire reason Class 3's skill for the reconciliation task didn't just paste both CSVs into one giant prompt and ask a single open-ended question, it broke the work into a five-part brief with explicit control totals, which is a context-engineering discipline this unit now gives a name and a research basis to, not just a habit that happened to work.

What this predicts, concretely, before you see it happen

Put the two findings together, attention as a fixed, thinning budget, and the U-shaped curve it produces, and a specific, testable prediction falls out.

That's not a claim to take on faith. It's a claim built to be tested, and the next unit tests it directly: a real transaction log, a real seeded error, moved to different positions inside it, run against models of different sizes, with the actual recall rates measured rather than assumed.

# a minimal illustration of softmax dividing a fixed budget
# across a growing number of tokens, the mechanism section above
# describes in words

import math

def softmax(scores):
    exp_scores = [math.exp(s) for s in scores]
    total = sum(exp_scores)
    return [e / total for e in exp_scores]

# same relevance score, competing against a growing number of others
short_context = softmax([2.0, 1.0, 1.0])            # 3 tokens
long_context = softmax([2.0] + [1.0] * 50)           # 51 tokens

print(f"attention on the relevant token, short context: {short_context[0]:.3f}")
print(f"attention on the relevant token, long context:  {long_context[0]:.3f}")
# the relevant token's own score never changed — only its share of a fixed budget did

Where this leaves you

You now have a mechanical account of context degradation, not just the warning from two classes ago but the actual cause: attention is a fixed budget divided by softmax across everything in the window, which produces a measured U-shaped recall curve favoring the start and end of a context over its middle, and which 2025's context-rot research shows compounding rather than dissolving as windows grow, specifically for the multi-part reasoning tasks Xolo's own reconciliation work is built from. Position in a document is no longer an incidental detail. It's a variable you now know how to reason about deliberately.

The next unit stops describing this effect and runs it live: a real needle-in-a-haystack demo, hunting one of Xolo's own seeded errors inside a synthetic transaction log built specifically to test where recall breaks down.