Test Whether Extra Context Helps Before You Buy a Longer Window
A bigger context window does not make every passage relevant. OpenAI's prompt engineering guide describes adding context as a way to give the model data it was not trained on and to constrain its answer to a set of resources you have chosen. It notes that context windows range from the low hundreds of thousands of tokens up to one million for newer GPT-4.1 models. Fitting more in is easy. Whether it helps is an experiment.
The protocol. Take 100 questions over a set of papers, each with one passage you have checked supports the answer. Run three conditions on the same questions:
- the gold passage alone;
- the gold passage plus 20 unrelated sections from other papers;
- the gold passage plus an earlier version of the same paper that reports a different number.
For each condition, count correct answers. In condition 3, also count answers that use the outdated number. Keep an "insufficient evidence" option and count how often the model uses it.
How to compare. The conditions share questions, so compare them pair by pair. Count b, the questions right alone and wrong with distractors, and c, the reverse. The comparison rests on b and c, not on the two totals; the book's evaluation chapter derives the paired test. For a single rate, the 95% interval is 1.96 × √(p(1 − p)/n): at p = 0.9 and n = 100 that is ±0.059.
What to expect. This site has not run this protocol, so it reports no result. If accuracy drops or the outdated number appears in condition 3, the fix is better selection, such as filtering by version and date or pruning retrieved passages, not a larger window. The book's context chapter treats what belongs in the window as an engineering budget.
