Managing Agent Context at Scale

Ask an agent "How much would catering cost for 160 people at £35 per head?" and give it a registry of six tools. Send every tool's schema to the model on every turn, the standard way, and answering that question costs 438 tokens. Have your own code look at the question first, notice the word "cost," and hand the model only the one relevant tool instead of all six, and the same answer costs 287 tokens — a 151-token, 34% cut, for zero change in the answer. That's this lesson's whole argument in one measured number: context window space is a budget, and most of what you send the model on any given turn, it never needed.

The question this lesson answers

Four techniques show up in this lesson that look, on the surface, like four different problems: recursive indexing over long documents, programmatic tool routing (the 34% cut above), skill loading, tool-output compression. They're not four problems. They're the same move, applied to four different budget lines in the same context window: decide what the model actually needs to see for this specific turn, and refuse to load the rest. The misconception this lesson corrects is reaching for the most drastic version of that move, summarisation, before trying the cheaper ones.

Recursive context: let the model build its own index

A student's research pointed at Recursive Language Models: instead of force-fitting a long document into one context window, the LLM builds a structured index over it, then navigates that index at query time.

This is not RAG. Vector retrieval works by embedding similarity — approximate, prone to returning something semantically close but factually wrong. A recursive index is built and navigated by the LLM itself, so it knows what each entry actually contains rather than guessing from a vector distance.

Two named prior-art citations anchor this as a real pattern, not a novelty: RAPTOR (2024) chunks, clusters, summarizes clusters into a tree, and traverses root to leaves at query time, outperforming flat RAG on multi-hop questions. MemWalker (2023) builds a navigation tree the LLM walks by relevance, with strong results past 100K tokens. Both are research-grade in 2024, production-viable in 2026 now that batch inference is cheap enough to build the index at all.

The Edinburgh venue-guide lab, measured

Six research notes on Edinburgh venues, summarized one-line-each, then queried:

def recursive_query(query: str, index: list[dict], chunks: list[dict]) -> str:
    # Step 1: LLM reads summaries and selects relevant chunk IDs
    index_text = "\n".join(f"[Chunk {e['id']}] {e['summary']}" for e in index)
    select_resp = client.chat.completions.create(
        model="meta-llama/Meta-Llama-3.1-8B-Instruct",
        messages=[{"role": "user", "content":
            f"Given these chunk summaries:\n{index_text}\n\nQuestion: {query}\n\n"
            f"Which chunk IDs are needed? Reply with ONLY comma-separated numbers."}],
        max_tokens=20, temperature=0)
    selected_ids = [int(x) for x in select_resp.choices[0].message.content.split(",") if x.strip().isdigit()]
    # Step 2: Load ONLY selected chunks into context
    selected_text = "\n\n".join(c["text"] for c in chunks if c["id"] in selected_ids)
    ...

python recursive_lab.py

The measured result: the model read six summaries, roughly 120 tokens, instead of six full chunks, roughly 600 tokens — a five-times reduction at six chunks, extrapolating to 50 times or more at 500. Selection was semantic, not word-overlap: "quiet room for a presentation" correctly mapped to the Haymarket Vaults chunk because the model reasoned about the match. Building the index is the expensive step, one call per chunk; querying it afterward is cheap and reusable — build once, query many times.

Programmatic tool calling: let code route, not the model

Standard function calling sends every tool schema to the model on every turn, and at scale that creates three named failure modes: schema bloat, where fifty tools at 200 tokens each burns 10,000 tokens before the user's actual message; selection confusion, where similar tools compete and the model picks the wrong one; and latency, from reasoning over every schema before acting at all.

Pydantic AI pioneered the alternative in 2025: instead of the model choosing which tools to consider, your code analyzes the request — keyword match, a classifier, or a fast LLM — selects the two or three relevant tools, and injects only those into context. The model still fills in parameters; it just never chooses the tool from a pile of fifty.

The measured head-to-head

A six-tool Edinburgh registry, tested both ways against the same query:

ROUTE_TABLE = {
    "venue": ["search_venues", "check_availability"], "pub": ["search_venues", "check_availability"],
    "capacity": ["check_availability"], "weather": ["get_weather"], "outdoor": ["get_weather"],
    "cost": ["calculate_cost"], "price": ["calculate_cost"], "catering": ["calculate_cost"],
    "transport": ["get_transport"], "bus": ["get_transport"], "train": ["get_transport"],
    "book": ["book_venue"], "confirm": ["book_venue"],
}

def programmatic_select(query: str) -> list[dict]:
    """Select tools by keyword match — zero LLM calls."""
    selected = {t for kw, tools in ROUTE_TABLE.items() if kw in query.lower() for t in tools}
    return [t for t in ALL_TOOLS if t["function"]["name"] in selected] if selected else ALL_TOOLS

Run against "How much would catering cost for 160 people at £35 per head?":

Thirty-four percent fewer tokens, because five unused schemas were never sent — the correct tool was selected in both cases, but standard reasoned over all six options where routing had exactly one. At fifty tools, the named extrapolation is roughly 80% savings, since schema overhead dominates as the count grows. The trade-off is real: routing removes the model's ability to surprise you with a creative tool combination. Use standard calling under ten tools, where judgment matters; use a router past twenty, where overhead dominates.

The skill loader: implementing what lesson 11 defined

Lesson 11 explained what a skill file is and its six-question anatomy. This is the code that consumes one.

SKILLS_DIR = Path("skills/")

def load_skill(skill_name: str) -> str:
    """Load a skill file by name. Returns empty string if not found."""
    skill_path = SKILLS_DIR / skill_name / "SKILL.md"
    return skill_path.read_text() if skill_path.exists() else ""

def build_context(system_prompt: str, task: str, relevant_skills: list[str]) -> list[dict]:
    """Build the message list with only relevant skills injected."""
    skills_text = "\n\n---\n\n".join(load_skill(s) for s in relevant_skills if load_skill(s))
    full_system = system_prompt + (f"\n\n<skills>\n{skills_text}\n</skills>" if skills_text else "")
    return [{"role": "system", "content": full_system}, {"role": "user", "content": task}]

The real worked CLAUDE.md for the Edinburgh agent names its tools with their paradigm from lessons 08 and 09 — search_venues as the CLI grep wrapper, check_availability as a function, get_weather as the Open-Meteo call — and states its rules in plain language: check availability before booking, never book without explicit confirmation, vegan options are mandatory, and a £6,000 total budget cap that rejects anything over it. book_venue is marked as requiring human approval — the destructive-tier rule from lesson 07, written directly into the project's own skill sheet.

:::tabs{labels="Claude Code (CLAUDE.md),OpenClaw (skills/ directory)}

One project-level file, loaded once per session, carrying identity, tool inventory, and global rules together in a single always-present document.

Each skill is self-contained, and the agent loads only the ones the current task needs — not the whole directory every time.

:::

Three layers now sit clearly separated: the system prompt defines who the agent is, tool schemas define what it can do, and skill files define how to do it well. Loading all three in full for every tool is the schema-bloat problem in a different costume. Load skills selectively, exactly the way the router above loads tool schemas selectively.

Compression, ranked by where it actually pays off

Turn 1 of a tool-using conversation costs roughly 2,000 tokens. By turn 5, system prompt plus accumulated history plus tool schemas plus reasoning routinely reaches 14,000 — and twenty tools at 200 tokens each is 4,000 tokens re-sent on every single turn, before the user's actual question is even read.

Layers one and two, attention cost and the KV cache, are the provider's responsibility, handled at training and inference time. Layers three through five — what reaches the model at all — are the agent engineer's, and this week's leverage sits mostly at layer three: raw input, before it ever reaches the model.

The highest-impact layer-three move: compress tool outputs before they re-enter context. A web search might return 5,000 tokens when the agent needs 200 of them.

def compress_tool_output(raw_output: str, query: str, max_tokens: int = 200) -> str:
    """Use a fast model to extract only query-relevant content."""
    resp = client.chat.completions.create(
        model="meta-llama/Meta-Llama-3.1-8B-Instruct",
        messages=[{"role": "user", "content":
            f"Extract ONLY the facts relevant to this question: {query}\n\n"
            f"Source text:\n{raw_output[:3000]}\n\nReply in under {max_tokens} tokens."}],
        max_tokens=max_tokens, temperature=0)
    return resp.choices[0].message.content.strip()

One extra 8B call, roughly 50ms, pays for itself within two turns against the 4,800 tokens it removes from every turn after.

The misconception: summarisation is not the first move

Here's the trap this lesson corrects directly. Faced with a bloated context window, the instinct is to reach for conversation summarisation first — pass history to a model, replace it with a summary. That's the most expensive, highest-risk move on the list, not the cheapest.

Stripping thinking tokens costs nothing in risk and removes text the user never saw regardless — the first move, not the last resort. Programmatic tool selection is the routing work above, already measured at 34%. Summarisation sits at the bottom on purpose: the most powerful lever and the lossiest one, and every major provider treats it that way. Claude auto-compacts around 95% of capacity server-side; Claude Code's manual /compact follows a tighter 70% rule, because summarizing already-degraded context produces worse summaries than summarizing sooner. OpenAI extracts facts across sessions rather than compressing within one; Gemini favors a frozen-prefix cache over summarizing at all. None of the three reaches for summarisation first, and neither should you.

Prompt caching closes the economic loop: frozen prefixes at a steep discount everywhere — Claude at $3.00 per million tokens standard versus $0.30 cached (90% off), GPT-4o at $2.50 versus $0.25–1.25 cached (50–90% off), Gemini at $2.00–4.00 versus $0.50–1.00 cached (50–75% off). Past 5,000 tokens of system prompt plus schemas and three calls per session, caching stops being optional — paying full price for the same tokens every turn is money left on the table for no quality gain.

Closing this thread, and the two it hands off

This lesson closes the implementation layer this week has been building toward: recursive indexing, programmatic routing, the skill loader, and the compression taxonomy all apply the same discipline — decide what this turn actually needs, and refuse to load the rest. One thread recurs here only as a bridge and stays owned elsewhere. The hallucination-versus-jailbreaking distinction and the banking demo's cascade failure are lesson 12's material in full — the chocolate-cake incident, grounding with tools, domain locking, all worked there; this lesson only reuses the same cascade incident to motivate why a pipeline needs sizing discipline in the first place. Persistent memory across sessions — what an agent remembers once the conversation ends, not just what fits in one window — is Week 4's subject, named here only as the roadmap this week's compression work sets up.