Context engineering: a budget for what the model sees

Draft chapter. The textbook guide covers setup and the supplied code. Practice in Exercise Book 21, then check Solutions 21 once you have tried it.

Chapter 20 ran Lucy's shop for a whole day. A long day makes a long conversation, and every chapter before it added something the model reads on every turn:

• the transcript (Chapter 3);
• remembered notes (Chapter 5);
• skills (Chapter 7);
• tool descriptions (Chapter 14);
• and every tool result the agent ever fetched.

Nothing so far owned the total. This chapter does.

Context engineering is deciding what the model reads on each call, within a budget. Part A measures what a long context does to answers, on small models you can run at home and on Claude: where a fact sits, what an estimate of its size gets wrong, what compaction keeps, what a long tool result is worth, and what the order of the prompt costs. Part B builds the context manager that sits between Lucy's loop and the model and keeps four promises:

• stable layers first;
• every token accounted for;
• the application's records never compacted;
• nothing cut silently.

Learning objectives

Part A measures a long context. After it you should be able to:

• measure how the position of a fact and the length of the context change whether a model finds it, and how that differs between a small local model and a frontier model;
• explain why a character-based token estimate fails for numbers, JSON and newer tokenizers, and what a server can do with a prompt longer than its window;
• compare ways of shortening a conversation by what the model can still answer, not by how many tokens they save;
• explain why a program-computed digest of a tool result can beat sending the whole result;
• measure what putting volatile text first costs on every turn, in prefill time on a local server and in dollars on an API.

Part B builds the context manager. After it you should be able to:

• keep a request in layers ordered by how often they change, and account for every token to one of them;
• spill an oversized tool result to a file and send a pointer that keeps both ends;
• compact a conversation in two steps that never touch the application's pinned records;
• refuse a context that cannot fit, instead of cutting it.

The deliverable is a context manager for Lucy's agent. It fits a long day into a budget, and it keeps her rules whatever a summary drops. The checkpoint tests it without a model, including against a summarizer that forgets everything, and it recomputes Part A's tables from both receipts, including what the Claude run cost.

Part A: what a long context does

The helpers live in the chapter's learner file. The measurements come from the chapter's experiment. The experiment ran each measurement twice on the same inputs:

• Small local models: qwen2.5:0.5b, qwen2.5:1.5b and qwen3:0.6b through Ollama 0.32.5 on the author's Mac mini, at temperature zero, with a 32,768-token window. Each model was unloaded before each measurement.
• Claude, through the Messages API: Claude Haiku 4.5 at temperature zero, and Claude Sonnet 5.5, which accepts no temperature setting and ran with up-front thinking turned off. The whole run cost $8.46 USD at list price. Its receipt records every token, and the checkpoint recomputes the cost from them.

The two groups answer different questions. The local models show what a long context does to a model small enough to run anywhere. Claude shows which of those failures a frontier model removes, and which ones no model removes.

import json
import runpy
from collections.abc import Callable, Sequence
from dataclasses import dataclass, field
from typing import Any

ctx = runpy.run_path("book/textbook/learner/profrod_sovereign_agent_ch21_context_learner.py")
# The listings below use these from the learner file.
Message, Spilled, ContextOverflowError = ctx["Message"], ctx["Spilled"], ctx["ContextOverflowError"]
estimate_tokens, message_tokens, head = ctx["estimate_tokens"], ctx["message_tokens"], ctx["head"]
measured = json.loads(open("docs/evidence/book-ch21/ch21-context-receipt-v1.json").read())
on_claude = json.loads(open("docs/evidence/book-ch21/ch21-context-claude-receipt-v1.json").read())

Where a fact sits, and how much surrounds it

The first measurement hides eight deliveries in a log of near-identical ones. Each line is a record_delivery result such as {"order":"A-6305","flavor":"mango","tubs":12,"at":"09:41"}. The model is asked, one order at a time, how many tubs arrived. The log is 1,000, 4,000, 8,000 or 16,000 tokens long, counted by the model's own tokenizer, and the eight facts sit at the top, a quarter of the way in, in the middle, three quarters in, or at the end. That makes forty questions per length.

The local models were not run past 16,000 tokens, half their window.

View full-size diagram

xychart-beta
    title "Right answers as the context grows"
    x-axis "Tokens in the context" ["1,000", "4,000", "8,000", "16,000"]
    y-axis "Share of 40 questions right" 0 --> 1
    line "Claude" [1, 1, 1, 1]
    line "qwen2.5:1.5b" [0.95, 0.7, 0.55, 0.4]
    line "qwen3:0.6b" [0.775, 0.875, 0.775, 0.325]
    line "qwen2.5:0.5b" [0.45, 0.35, 0.275, 0]

For the small models, accuracy falls long before the window is full. At 16,000 tokens, half the window, the best of the three found fewer than half the facts. Where the facts sat mattered as much as how many tokens surrounded them:

View full-size diagram

xychart-beta
    title "qwen2.5:1.5b at 16,000 tokens, by position"
    x-axis "Where the facts sat" ["Top", "Quarter", "Middle", "Three quarters", "End"]
    y-axis "Share of 8 found" 0 --> 1
    bar [0.875, 0.125, 0.25, 0.375, 0.375]

The misses were not blanks. The 1.5b model answered with numbers from other deliveries: plausible, wrong and hard to catch. The 0.5b model failed differently. At 16,000 tokens it stopped following "answer with the number only", and every reply began to explain instead. At 1,000 tokens it had answered directly.

Both Claude models answered all 240 questions correctly: every length up to 150,000 tokens and every position, including the middle. On this task, finding one record among near-identical ones, losing facts in a long context is a limit of small models, not of long contexts. A frontier model does not make a long context free, though. Every token is paid for on every call. At 150,000 tokens, one question cost $0.15 USD on Haiku 4.5 and $0.30 USD on Sonnet 5.5 before caching. The rest of Part A finds failures that a larger model does not remove.

An estimate is not a count

Budgets are usually written in tokens and checked with an estimate, because counting exactly needs the model's tokenizer. The usual estimate is four characters per token. The experiment compared it with three tokenizers' own counts on four kinds of text: qwen2.5's, and Claude's, through the API's free counting endpoint.

View full-size diagram

xychart-beta
    title "Characters per token, by kind of text"
    x-axis "Kind of text" ["Prose", "Python", "JSON", "Sales log"]
    y-axis "Characters per token" 0 --> 6
    line "Estimate" [4, 4, 4, 4]
    line "qwen2.5" [5.09, 4.44, 2.46, 1.70]
    line "Haiku 4.5" [4.93, 3.90, 2.79, 1.77]
    line "Sonnet 5.5" [3.22, 2.85, 1.90, 1.56]

The estimate is roughly right for prose on two of the three tokenizers, and badly wrong for exactly what an agent's context fills with: tool results full of numbers and JSON. On the third it is wrong for everything. Sonnet 5.5 uses a newer tokenizer that cuts text into smaller pieces. The same prose is about half as many tokens again as on Haiku 4.5, and four characters per token undercounts every kind of text. A budget in tokens is a budget for one tokenizer: the second turn of the layout measurement below is 5,960 tokens on Haiku 4.5 and 8,745 on Sonnet 5.5. The first trial run of this very experiment made that mistake. It sized its longest log by the estimate, sent about 25,000 real tokens instead of 16,000, and overflowed the window it had set.

What happens at the window's edge was then measured on purpose. The 16,000-token log was sent to each model with an 8,192-token window. Ollama read 4,098 tokens, about half the window, and answered from those. It dropped the start of the log, where the facts were, and returned no error. All three models answered 0 of 8 and said there was no such order. A prompt longer than the window can be cut without anyone being told.

The Claude API does the opposite. Sent a 210,062-token log, 5% over Haiku 4.5's 200,000-token window, it refused the request with an error: prompt is too long: 210098 tokens > 200000 maximum. No answer came back. A loud refusal is better than a silent cut, but it is still a failure discovered at run time, after the prompt was built and sent. The counting endpoint had reported the size before anything was sent. Either way, a context manager must know the size of what it sends, and refuse rather than hope.

What compaction keeps

The second measurement is a long day in Lucy's own words. Early on she:

• names the shop;
• rules out ordering mango from Frostline;
• sets the closing time at 6 pm and the chocolate approval at 12 tubs;
• moves vanilla to Polar Dairy.

Later she changes the closing time to 7 pm and the approval to 15 tubs, and names the health inspector. In between come freezer checks and sales tables. The whole day is 6,398 estimated tokens. Clearing all but the latest tool result leaves 716.

Ten questions test it:

• five about rules and names;
• two about values she later changed;
• two answered only in tool results;
• one about the last thing she said.

The day was shortened in five ways, and each model answered the same questions:

Token counts here are the chapter's own estimate, so the columns compare the same content.

Six things stand out.

Clearing tool results cost almost nothing. It cut the day by 89%, and two of the three small models answered more questions, not fewer. The freezer and sales tables were mostly noise for these questions. Both Claude models lost exactly one answer: the tubs in order A-4817, which existed only in a cleared tool result. Clearing loses what it clears and nothing else, and the agent can fetch it again.

A naive window fails on every model. Keeping the most recent messages that fit, the usual first attempt, kept only the last few messages, 73 tokens. Just before them came a 140-row sales table, too large to fit, and a window stops at the first message that does not fit. Both Claude models answered 1 of 10 from it, no better than the small ones. No model can answer from what was not sent.

The whole day was the best context only for a capable model. With everything in front of it, qwen2.5:1.5b answered both changed values with the old ones, 6 pm and 12 tubs, and so did qwen3. The new values were in the context, later in the day. They were not what the models used. Both Claude models answered all ten from the whole day, including 7 pm and 15 tubs. Reading a long day correctly is something a larger model does better. Paying for 6,398 tokens on every turn is not.

A summary is only as good as the model that writes it. Asked to keep "every rule, decision, name and number Lucy stated, and where she changed something only the latest value", qwen2.5:1.5b wrote a summary that lost the inspector and the chocolate approval. It turned Lucy's rule into its opposite: "updates on orders from Frostline (mango)". It also mixed the two closing times. Claude's summaries kept every rule and both latest values (Sonnet 5.5 wrote "Closing time today: 7 pm (changed from 6 pm)"), in contexts of 190 and 287 tokens, and scored 9 of 10. A good summary is still a model's output on one day. It is a step you test on every change of model, not one you verify once.

View full-size diagram

xychart-beta
    title "How much the result depends on the model"
    x-axis "How the day was shortened" ["Whole day", "Recent", "Cleared", "Summary", "Pinned"]
    y-axis "Correct of 10" 0 --> 10
    line "Best model" [10, 3, 9, 9, 9]
    line "Worst model" [4, 1, 4, 1, 8]

Pinned records were the most reliable. With Lucy's rules and current values held as records at the top of the prompt, all three small models answered 8 of 10 at the 600-token budget, and all three got both changed values right. The records say "7 pm (changed from 6 pm)". The transcript only says both, at different times.

Only pinned records did not depend on the model. Across five models, from 0.5 billion parameters to Sonnet 5.5, pinned records scored 8 or 9 of 10 every time. The model's own summary ranged from 1 to 9 on the same day, and so did the whole day in context, from 4 to 10. A context manager cannot choose the model it will run with next year. It can choose to make the facts it relies on independent of the model.

A tool result is data, and a program can read it

The third measurement sends one tool result: a sales log of 480 lines, such as 14:07 | mango | 2 tubs | $7.00. It is 4,016 tokens by the estimate, and about 9,500 by the measured ratio for this kind of text. It was sent four ways:

• whole;
• cut to its first lines;
• cut to both ends;
• replaced by a 66-token digest a program computed from all 480 lines: the count, tubs by flavor, and the first and last sale.

There are five questions:

• total pistachio;
• the first sale;
• the last sale;
• the sale at 14:07;
• how many sales.

No model totaled the pistachio sales from the whole log, and none counted its lines, Claude included. Sending 480 lines did not mean the model read 480 lines. qwen3 did not even read them as sales: it answered with the price column, "7.00", to questions about flavors. Haiku 4.5 answered 147 tubs of pistachio when the log held 81, and 500 sales when it held 480. Sonnet 5.5 began adding the entries one by one inside its 48-token answer, and gave no total. A model asked to add up 480 lines in its answer is being asked to do a program's job.

The digest won on every model, with 2% of the tokens, because a program had already done the arithmetic that Chapter 3 found models doing badly. But a digest answers only the questions it anticipated: none of them could say what sold at 14:07. The models differed in whether they said so. Asked about 14:07, the small models named a flavor every time, even from the digest, which does not mention 14:07. Sonnet 5.5 said each time that what it had been given did not show 14:07, "The log ends at 12:52", and it named the right flavor from the whole log. Haiku 4.5 said so for the first lines and the digest, but named wrong flavors from both ends and from the whole log. A model that reports a gap in its view is easier to build on than one that fills it. Even a correct digest can be misread by a small model: qwen2.5:1.5b answered 100 when the digest said 81.

Put what changes last

The last measurement is Chapter 9's prefix cache, seen from the layout of the prompt. Ten turns of a conversation carry the same 1,500-token set of shop rules and a freezer snapshot that changes every turn. In one layout the snapshot is at the top of the system prompt. In the other, the rules come first and the snapshot goes in the latest message:

The same content cost eight to ten times more on every turn when the part that changes came first. A prefix cache reuses work up to the first difference. A timestamp at the top of a system prompt is a difference on every turn.

On the Claude API the difference is money. The API caches a prompt prefix when asked, bills writing it at 1.25 times the input price and bills reading it at a tenth, and reports both per request. Haiku 4.5 caches nothing shorter than 4,096 tokens, so the Claude run used 160 rules in place of 40. The run marked each request for caching:

View full-size diagram

xychart-beta
    title "Cost per turn with caching on, in cents"
    x-axis "Model, and what comes first" ["Haiku volatile", "Haiku stable", "Sonnet volatile", "Sonnet stable"]
    y-axis "Cents per turn" 0 --> 2.5
    bar [0.75, 0.07, 2.19, 0.21]

Stable first was ten times cheaper per turn on both models. Volatile first was worse than not caching at all: every turn wrote a new cache entry and no turn read one, so it paid 1.25 times the input price for nothing. Without caching, the same Haiku turn costs about $0.006 USD. Haiku's times were the same either way, about 0.8 seconds, because an API's time to answer includes the network and its queue. The saving you can count on is the bill.

The decision

Part A turns into five rules for Lucy's agent:

1. Count what you send, with the model's own tokenizer, and refuse what does not fit. A local server cut a long prompt without saying so. The Claude API refused one loudly. Either way, the manager should know first, and the same text is a different number of tokens on different models.
2. Keep the context short even when the window is large. Small models lost half the facts well inside their window, most of all in the body of a long context. Claude did not lose them to 150,000 tokens, but every model pays for every token on every call.
3. Clear old tool results first. They are most of the tokens, and the agent can fetch them again.
4. Keep rules and current values as records the application owns, not as sentences in the transcript. A summary, even a good model's own, is not a store. Pinned records were the only strategy that scored well on every model.
5. Let a program read large results, and put what changes last. No model totaled a 480-line log, and on an API the order of the prompt decides whether its stable part costs a tenth of the input price or a quarter more.

Part B builds a context manager that follows all five.

Part B: a context manager for Lucy's agent

Part A measured what a long context does. This part builds the component that decides what goes into it. Every earlier chapter added to what Lucy's model sees: Chapter 3's transcript, Chapter 5's memory, Chapter 7's skills, Chapter 14's tools, and every tool result in between. Until now nothing owned the total. The context manager does. It sits between the agent loop and the model call and keeps four promises:

• Order: stable layers go first and volatile ones last, so the prefix cache keeps working.
• Accounting: every token belongs to a layer, and the total is checked against a budget.
• Records outlive the transcript. The application's pinned records are never compacted.
• No silent cuts: a context that cannot fit raises an error; it is never cut somewhere nobody chose.

Load the learner file:

import json
import runpy
import tempfile
from pathlib import Path

ctx = runpy.run_path("book/textbook/learner/profrod_sovereign_agent_ch21_context_learner.py")
Layers = ctx["Layers"]

Layers, stable first

A request has three layers, ordered by how often they change. The system text changes almost never. The pinned records are facts the application owns: Lucy's rules, approvals and suppliers, held in the Chapter 4 store and changed only through Chapter 11's approval path. They change rarely. The conversation grows every turn. Layers keeps them apart, and budget_report counts each one, as /context does in a coding agent:

@dataclass
class Layers:
    """Everything the model will see, in the order it is sent: stable first, volatile last."""

    system: str
    pinned: list[str] = field(default_factory=list)
    conversation: list[Message] = field(default_factory=list)

    def messages(self) -> list[Message]:
        """The request's messages. The system text and pinned records never move."""
        head_text = self.system
        if self.pinned:
            head_text += "\n\nRecords Lucy's shop keeps (current):\n" + "\n".join(
                f"- {record}" for record in self.pinned
            )
        return [{"role": "system", "content": head_text}, *self.conversation]


def budget_report(
    layers: Layers, estimate: Callable[[str], int] = estimate_tokens
) -> dict[str, int]:
    """Tokens per layer, like `/context` in a coding agent."""
    pinned = sum(estimate(record) for record in layers.pinned)
    conversation = sum(message_tokens(m, estimate) for m in layers.conversation)
    system = estimate(layers.system)
    return {
        "system": system,
        "pinned": pinned,
        "conversation": conversation,
        "total": system + pinned + conversation,
    }

day = Layers(
    "You are the assistant for Lucy's ice cream shop.",
    ["Never order mango from Frostline.", "Vanilla is ordered from Polar Dairy."],
    [
        {"role": "user", "content": "Check the freezer."},
        ctx["tool_message"]("stock_level", {"freezer": {"vanilla": 12, "mint": 3} | {f"f{i}": i for i in range(60)}}),
        {"role": "assistant", "content": "Mint is low."},
    ],
)
print(ctx["budget_report"](day))
print(day.messages()[0]["content"])

{'system': 12, 'pinned': 18, 'conversation': 159, 'total': 189}
You are the assistant for Lucy's ice cream shop.

Records Lucy's shop keeps (current):
- Never order mango from Frostline.
- Vanilla is ordered from Polar Dairy.

The records render inside the system message, after the fixed instructions. They are the second layer of the request, and they change only when a record changes.

Count with a margin

estimate_tokens divides characters by four. Part A measured that on qwen2.5 it overcounts prose by 27% and undercounts numbers by 57%, and that on Sonnet 5.5's tokenizer it undercounts every kind of text. So a budget built on it must sit well inside the model's window, or use the tokenizer's own count where the server offers one. Part A also showed the cost of getting it wrong: a server that receives more than its window can cut the prompt without an error.

A tool result too large to send

Part A measured that sending a whole 480-line log did not make any model read it well, Claude included. When a tool result is too large, the manager sends both ends of it and a pointer to the whole. It writes the full output to a file, so the agent can read a part of it on purpose instead of losing it without knowing:

def head_and_tail(text: str, budget: int, estimate: Callable[[str], int] = estimate_tokens) -> str:
    """Whole lines from both ends, half the budget each, with a marker for what was left out."""
    lines = text.splitlines()
    first = head(text, budget // 2, estimate).splitlines()
    last: list[str] = []
    used = 0
    for line in reversed(lines[len(first) :]):
        cost = estimate(line + "\n")
        if used + cost > budget // 2:
            break
        last.insert(0, line)
        used += cost
    omitted = len(lines) - len(first) - len(last)
    if omitted <= 0:
        return text
    return "\n".join([*first, f"[... {omitted} lines omitted ...]", *last])


def spill_tool_result(
    text: str,
    limit: int,
    folder: Path,
    name: str,
    estimate: Callable[[str], int] = estimate_tokens,
) -> Spilled:
    """Send a tool result whole if it fits in `limit`; otherwise save it and send a pointer.

    The pointer keeps both ends of the output and says where the whole output is, so the agent can
    read a part of it on purpose instead of losing it without knowing.
    """
    tokens = estimate(text)
    if tokens <= limit:
        return Spilled(text, None, tokens)
    folder.mkdir(parents=True, exist_ok=True)
    path = folder / f"{name}.txt"
    path.write_text(text)
    note = f"[output of {tokens} tokens saved to {path}; showing both ends]"
    shown = head_and_tail(text, max(limit - estimate(note) - 8, 0), estimate)
    return Spilled(f"{note}\n{shown}", path, tokens)

log = "\n".join(f"{10 + i // 60}:{i % 60:02d} | vanilla | 2 tubs | $7.00" for i in range(400))
folder = Path(tempfile.mkdtemp(prefix="ch21-"))
spilled = ctx["spill_tool_result"](log, 120, folder, "sales_today")
print(spilled.original_tokens, "tokens ->", ctx["estimate_tokens"](spilled.text), "tokens sent")
print(spilled.text.replace(str(folder), "<folder>"))

3300 tokens -> 103 tokens sent
[output of 3300 tokens saved to <folder>/sales_today.txt; showing both ends]
10:00 | vanilla | 2 tubs | $7.00
10:01 | vanilla | 2 tubs | $7.00
10:02 | vanilla | 2 tubs | $7.00
10:03 | vanilla | 2 tubs | $7.00
[... 392 lines omitted ...]
16:36 | vanilla | 2 tubs | $7.00
16:37 | vanilla | 2 tubs | $7.00
16:38 | vanilla | 2 tubs | $7.00
16:39 | vanilla | 2 tubs | $7.00

The marker in the middle says how many lines were left out. A model that sees "392 lines omitted" knows its view is partial. A model that sees only the first ten lines of a log may take them for the whole day.

Compaction in two steps

When the conversation outgrows the budget, compact works in two steps.

The first step clears old tool results. The tool calls stay visible with a note of how large each result was, so the agent knows it looked something up and can look it up again. In Part A, this step alone cut Lucy's day by 89%, so it usually fits the budget without a summary.

The second step, only if the first was not enough, replaces everything before the latest question with a summary. The pinned records and the system text are never touched:

def clear_old_tool_results(messages: Sequence[Message], keep_last: int) -> list[Message]:
    """Replace the content of all but the last `keep_last` tool results with a short marker.

    The call that produced each result stays in the conversation, so the agent can see that it
    looked something up, and can look it up again.
    """
    tool_positions = [i for i, m in enumerate(messages) if m.get("role") == "tool"]
    old = set(tool_positions[: max(len(tool_positions) - keep_last, 0)])
    cleared: list[Message] = []
    for i, message in enumerate(messages):
        if i in old:
            size = estimate_tokens(str(message.get("content", "")))
            cleared.append(
                {**message, "content": f"[tool result cleared to save context: {size} tokens]"}
            )
        else:
            cleared.append(dict(message))
    return cleared


def compact(
    layers: Layers,
    budget: int,
    summarize: Callable[[list[Message]], str],
    keep_tool_results: int = 2,
    estimate: Callable[[str], int] = estimate_tokens,
) -> tuple[Layers, dict[str, Any]]:
    """Fit `layers` into `budget` in two steps, and report what each step did.

    1. Clear old tool results, keeping the last `keep_tool_results`.
    2. If still over, replace the oldest turns with one summary message, keeping the latest user
       message and everything after it.

    The system text and pinned records are never touched. If they alone do not fit, or the latest
    turn does not fit beside them, raise `ContextOverflowError`.
    """
    report: dict[str, Any] = {"before": budget_report(layers, estimate)["total"], "steps": []}
    fixed = Layers(layers.system, list(layers.pinned), [])
    if budget_report(fixed, estimate)["total"] > budget:
        raise ContextOverflowError("the system text and pinned records alone exceed the budget")
    current = Layers(layers.system, list(layers.pinned), [dict(m) for m in layers.conversation])
    if budget_report(current, estimate)["total"] <= budget:
        report["after"] = report["before"]
        return current, report
    current.conversation = clear_old_tool_results(current.conversation, keep_tool_results)
    report["steps"].append(
        {"step": "clear-tool-results", "total": budget_report(current, estimate)["total"]}
    )
    if budget_report(current, estimate)["total"] <= budget:
        report["after"] = report["steps"][-1]["total"]
        return current, report
    last_user = max(
        (i for i, m in enumerate(current.conversation) if m.get("role") == "user"),
        default=len(current.conversation),
    )
    recent = current.conversation[last_user:]
    older = current.conversation[:last_user]
    if not older:
        raise ContextOverflowError("the latest turn alone exceeds the budget")
    summary = {
        "role": "user",
        "content": "Summary of the earlier conversation:\n" + summarize(older),
    }
    current.conversation = [summary, *recent]
    total = budget_report(current, estimate)["total"]
    report["steps"].append({"step": "summarize", "turns": len(older), "total": total})
    if total > budget:
        raise ContextOverflowError(
            f"after compaction the context still needs {total} of {budget} tokens"
        )
    report["after"] = total
    return current, report

Here is a conversation in which Lucy states a rule once and then asks for six freezer checks. The summarizer below is deliberately bad: it remembers only that there were freezer checks. Watch the rule at three budgets:

rule = "Never order mango from Frostline again; they were late twice."
talk = [{"role": "user", "content": rule}, {"role": "assistant", "content": "Noted."}]
for i in range(6):
    talk += [
        {"role": "user", "content": f"Check the freezer ({i})."},
        ctx["tool_message"]("stock_level", {"rows": ["vanilla 12"] * 40}),
        {"role": "assistant", "content": "Here it is."},
    ]
talk.append({"role": "user", "content": "Can we order mango from Frostline?"})
layers = Layers("You are the assistant for Lucy's ice cream shop.", [], talk)
forget = lambda older: "(earlier: freezer checks)"  # noqa: E731
for budget in (2000, 600, 150):
    compacted, report = ctx["compact"](layers, budget, forget)
    visible = any(rule in m["content"] for m in compacted.messages())
    print(budget, json.dumps(report["steps"]), "rule visible:", visible)

2000 [] rule visible: True
600 [{"step": "clear-tool-results", "total": 495}] rule visible: True
150 [{"step": "clear-tool-results", "total": 495}, {"step": "summarize", "turns": 20, "total": 45}] rule visible: False

At 2,000 tokens nothing was needed. At 600, clearing the tool results was enough, and the rule was still there in Lucy's own words. At 150 the summary replaced it, and the rule was gone. Part A's real summaries did better than this one: Claude's kept every rule, and a small model's inverted one. A context manager cannot know in advance which kind it will get.

Now keep the rule where it belongs, as a pinned record, and compact to the same 150 tokens with the same bad summarizer:

pinned = Layers("You are the assistant for Lucy's ice cream shop.", ["Never order mango from Frostline."], talk)
compacted, report = ctx["compact"](pinned, 150, forget)
print(report["after"], "tokens")
for message in compacted.messages():
    print(f"{message['role']:9} {message['content'][:90]!r}")

54 tokens
system    "You are the assistant for Lucy's ice cream shop.\n\nRecords Lucy's shop keeps (current):\n- N"
user      'Summary of the earlier conversation:\n(earlier: freezer checks)'
user      'Can we order mango from Frostline?'

The rule is in the system message, which compaction never touches. The question is still there, and the context fits in 54 tokens. This is the chapter's design rule: a fact the application must rely on is a record, not a sentence in the transcript. Compaction can shorten a conversation. It must never be the only thing between Lucy and her own rules.

Refuse what cannot fit

compact raises ContextOverflowError when the system text and pinned records alone exceed the budget. It also raises when the latest turn cannot fit beside them, and when a summary still leaves the context too large:

try:
    ctx["compact"](Layers("s", [], [{"role": "user", "content": "x" * 2000}]), 150, forget)
except ctx["ContextOverflowError"] as error:
    print("refused:", error)

refused: the latest turn alone exceeds the budget

A refusal is a failure someone can see: the agent can ask for a narrower tool result, split the question, or report that it cannot answer. A silent cut is a wrong answer later.

Where the manager sits

In Chapter 3's loop, the runtime appended each tool result to the transcript and sent the whole transcript on every turn. With a manager, each turn does four things before the model call:

1. Assemble Layers from the system text, the current pinned records read from the store, and the conversation.
2. Spill any new tool result over its limit.
3. Compact if the total exceeds the budget.
4. Send the result of messages(), which is stable first, so the prefix cache from Part A's layout measurement can reuse it, on a local server or an API.

The transcript on disk, from Chapter 4 and Chapter 8, stays complete. Compaction changes what the model is shown, not what the shop has recorded.

In a coding agent

Claude Code's context management follows the same order you built. It "clears older tool outputs first, then summarizes the conversation if needed", and it warns that "detailed instructions from early in the conversation may be lost". The documentation draws this chapter's conclusion: "Put persistent rules in CLAUDE.md rather than relying on conversation history." You can steer a summary with a "Compact Instructions" section in CLAUDE.md, or with /compact and a focus. /context shows what fills the window. If a single output is so large that the context refills after every summary, Claude Code stops auto-compacting after a few attempts and shows an error instead of looping.

Its prompt caching uses the same layers in the same order: the system prompt, then project context such as CLAUDE.md, then the conversation, "so content that rarely changes between turns comes first". Tool results get the same treatment as your spill_tool_result. When an MCP tool's result exceeds the output limit, Claude Code saves it to a file and puts the path in the conversation, so Claude reads the file when it needs the content.

Try it in Claude Code: in a long session, run /context, then /compact with a focus, then /context again. Find which of your early instructions are still in the summary, and move the ones that must survive into CLAUDE.md.

Expected observations and learner verification

Run the chapter's checkpoint from the course folder:

uv run python book/textbook/checkpoints/profrod_sovereign_agent_ch21_context_checkpoint.py

It prints six lines, each beginning ok:

• estimates, depths and both cuts stay inside their budgets;
• an oversized tool result is saved whole and replaced by a pointer that fits;
• stable layers come first, and every token is accounted to a layer;
• compaction clears tool results first, then summarizes, and only pinned rules survive a summarizer that forgets everything;
• a context that cannot fit is refused, never cut silently;
• both receipts' tables recompute from their rows, the inputs regenerate identically, and the Claude run's cost recomputes from its tokens at list price.

The checkpoint needs no model. To repeat Part A locally, pull the three models into Ollama and run the experiment with --out pointing at a new file. Expect the token counts and the layout's pattern to match. Accuracy should match within a question or two per cell, and the times will differ with your hardware.

To repeat the Claude run, set ANTHROPIC_API_KEY, or put it in the course folder's .env, which git ignores. Then add --provider anthropic --max-usd 12 and a new --out file. The run stops before starting a request once its spending reaches the ceiling. Expect about $8.50 USD, and a question or two of difference per cell on Sonnet 5.5, which runs without a temperature setting.

Exercises

Exercise 1: find a long-context task a frontier model fails

Claude found every delivery at every length up to 150,000 tokens, but that task needs one record. Change the question so it needs many: "how many tubs came from orders placed before 10:00?" Run it on a small model and on Claude at 1,000, 16,000 and 64,000 tokens. Where does each fail, and does a program's digest fix it?

Exercise 2: summarize with a schema

Replace the free-text summary with one that must fill fixed fields: rules, decisions with their latest values, names and open questions. Use the constrained output from Chapter 2. Score it on the same ten questions. Does a schema stop the model from keeping superseded values?

Exercise 3: a query tool for spilled output

Give the agent a read_spilled(path, pattern) tool that returns the matching lines of a spilled tool result, bounded like everything else. Ask Part A's truncation questions again. Which questions does it answer now, and which still need a program?

Exercise 4: place a second cache breakpoint

The Claude run cached with one automatic breakpoint at the end of each request. Keep the volatile snapshot at the top of the system prompt, but mark an explicit breakpoint after the stable rules instead. What share of each turn is read from the cache now, and what does a turn cost? Compare the reported reads with cached_prefix_tokens and explain any difference.

Exercise 5: budget by kind of text

Replace estimate_tokens with an estimate that uses Part A's measured ratio for each kind of text in the request. How much closer does the budget come to the tokenizer's count on Lucy's day, and how large a margin do you still need?

Exercise 6: decide what to pin

List every fact in Lucy's day that the agent relied on in Part A's questions. For each, decide whether it belongs in a pinned record, in the store behind a tool, or only in the conversation. Write the rule you used to decide, and test it on a new day.

Active recall

Without rereading: where in a long context did the models find facts most reliably, and where least? What did the local server do with a prompt longer than its window, and what did the Claude API do? How far is four characters per token from the tokenizer for prose, and for numbers, and which model's tokenizer did it undercount everywhere? What share of Lucy's day was tool results?

Which step of compaction comes first, and why? What must compaction never touch? What does a pointer to a spilled tool result give the agent that a cut does not? Why does a timestamp at the top of the system prompt cost a full read on every turn?

Vocabulary

The context is everything a model reads for one call. Its window is the most it can read, and its budget is the most the application decides to send. A layer is part of the context grouped by how often it changes: system text, pinned records that the application owns, and the conversation. Compaction fits a conversation into a budget, first by clearing old tool results and then by summarizing the oldest turns. Spilling saves an oversized tool result to a file and sends a pointer. A prefix cache reuses the model's work on the unchanged start of a request. Silent truncation is a server cutting a prompt that exceeds its window without reporting an error.

Summary

A long context costs more than tokens, and how much more depends on the model. Well inside a 32,768-token window, the best of three small models went from finding 38 of 40 facts at 1,000 tokens to 16 of 40 at 16,000, answering with numbers from look-alike records. Claude Haiku 4.5 and Sonnet 5.5 found all 240 at every length to 150,000 tokens. Four characters per token overcounted prose on two tokenizers and undercounted everything on Sonnet 5.5's. Sent a prompt twice as long as its window, the local server read about a quarter of it and answered from that, with no error. The Claude API refused an over-window prompt outright.

Clearing old tool results cut Lucy's day by 89% at almost no cost in answers. A small model's own summary lost facts and inverted a rule, while Claude's kept every rule. The whole day in context led two small models to use Lucy's old closing time, and Claude to the right one. Only one strategy scored well on every model: her rules and current values held as records the application owns, 8 or 9 of 10 in 602 tokens. No model totaled a 480-line sales log, and a program's 66-token digest beat sending it on all five. Putting the part that changes last cut local prefill eight- to tenfold and a Claude turn's cost tenfold.

You built the component that owns the total: layers ordered stable first, a budget report for each, spilled tool results with a pointer, compaction that clears before it summarizes, pinned records that no compaction touches, and a refusal instead of a silent cut. Lucy's agent now decides what its model reads.

Keep building with Prof Rod

Found this material through a colleague, classroom or shared download? Get the complete book at profrod.ai/book and join the Prof Rod learner community. Bring one result, one question or one failure you learned from. Share this resource with another learner and keep its source links with it so they can find the full course and future updates.