Read an Agent Architecture Diagram by Tracing Every Arrow to Code

Teaching fixture: Eval sets, token counts and prices are invented so the arithmetic is checkable. Money is USD, stored as integer cents.

The diagram that lied about where the budget came from

A design doc for an eval agent shows a rectangle labeled "trusted config" between the planner model and the run launcher. The arrow into that rectangle is drawn from the model's output. There is no second arrow from a config file. Trace the code instead of the picture and you find the launcher reading budget_cents from the plan object the model just produced. The label says trusted. The wire says otherwise.

This is the failure worth building a habit against: a diagram that records an intention rather than a data flow. A diagram should be a map of where values come from and where they go. When a box's label and its inputs disagree, the diagram misleads everyone who reads it instead of the code. So the question to ask is not "does this look like a sensible system?" but "does every arrow name a real value and its source of authority?" The second question can be checked. The first cannot.

Anthropic's writing gives useful vocabulary. Its piece on managed agents separates session history, the harness and the execution environment into distinct components, which is a good prompt for asking which component owns which value in your own diagram. Its piece on writing tools for agents notes that how a tool is described changes how an agent uses it, so the interface shapes what the model proposes. Authority still comes from permissions and checks your code enforces. Neither piece draws your diagram for you.

What authority means on an arrow

Two terms first. A trusted value is one your application got from a source it designated in advance: controlled config, an authorized registry read, a signed manifest whose provenance was checked. Storage location alone does not confer authority; a database can hold untrusted text. A generated value is anything a model produced, however well formed. The distinction is provenance, not quality. A model-written budget can be numerically correct and still carry no authority, because nobody with authority set it.

Each arrow into a decision should carry two facts: which value moves along it, and which of those two kinds it is. The label on a box is a claim about what the box does. The arrows feeding it are the evidence.

A worked case: the eval-plan validator

Here is a system small enough to trace completely. An agent in a research repo proposes the next evaluation run of a prompt change. The application owns a config: an eval registry (reasoning-v3 has a 400-case dev split and a 1,000-case test split), a split rule (iteration may only use dev), a sampling policy (5 samples per case), a token estimate (1,200 input and 300 output tokens per sample), prices (300 and 1,500 cents per million tokens, invented for the fixture) and a budget of 2,000 cents.

The reference, computed from config alone: 400 × 5 = 2,000 samples; 2.4 million input tokens at $3.00 per million is $7.20; 0.6 million output tokens at $15.00 per million is $9.00; total $16.20, under $20.00. That arithmetic is the oracle. The model cannot change it by proposing something else. A companion article builds and tests the full check.

Now draw it. A naive diagram shows four boxes in a line: "Config", "Model", "Validator", "Accepted plan". The line hides the only question that matters: which arrow into the validator carries budget_cents, and which carries the proposed runs? If both come from the same place, the diagram is wrong. Here is the reading excerpt the honest diagram has to match. It assumes validated Plan and EvalConfig models, duplicate and unknown-set checks upstream, and a PolicyError type; it is for tracing, not a standalone program.

def validate_plan(raw_json, config):
    plan = Plan.model_validate_json(raw_json)
    if plan.config_id != config.config_id:
        raise PolicyError("stale_config")
    if any(run.split not in config.allowed_splits for run in plan.runs):
        raise PolicyError("split_not_allowed")
    proposed = {run.eval_set: run.n_cases for run in plan.runs}
    required = {name: config.registry[name].dev for name in config.requested}
    if proposed != required:
        raise PolicyError("case_count_mismatch")
    cost = config.cost_cents(sum(required.values()) * config.samples_per_case)
    if cost > config.budget_cents:
        raise PolicyError("budget_exceeded")
    return {"status": "accepted_draft", "estimated_cost_cents": cost}

config is a separately validated object loaded from the repo's controlled config. plan is built from whatever JSON the model produced. budget_cents, prices and the split rule live only on config. The Plan schema has no such fields, and extra="forbid" in its Pydantic configuration rejects any attempt by the model to add one. That is the code-level fact the diagram must show as two separate arrows into the validator, meeting at the comparison rather than merging before it.

View full-size diagram

The cover repeats the article's title. The field-level figure below adds the exact values a reviewer should find in the code.

View full-size diagram

Building a dictionary from plan.runs can hide a duplicated run before the comparison. The excerpt relies on an upstream uniqueness check; confirm it exists as part of the trace rather than assuming the short function shows every boundary. Likewise, the absence of budget_cents from the schema only helps if the parse call actually applies that schema to the object used later.

Test the diagram with seven failure cases

A diagram earns trust the way a claim does: by surviving attempts to break it. Seven narrow cases, each isolating one rule, show which arrow a failure belongs to. A failing case narrows the search; it does not by itself identify the defective line.

Each maps to an arrow. The stale-config case tests the config_id comparison, which exists because the plan must name the config version it was drafted against; matching IDs does not prove either is the newest version. The self-approved-budget case tests whether the model can smuggle a budget through the plan at all. It cannot, because Plan has no field to carry it. If someone removed extra="forbid", Pydantic's default would silently ignore the extra field; the explicit rejection rule would break, but budget authority would not move, because the function still reads config.budget_cents. Change that one read to plan.budget_cents and the authority arrow does move, while the picture stays the same. That is how a diagram becomes false without anyone touching it.

Exact match and budget are two boxes, not one

The validator does not accept "close enough". It requires proposed == required, an exact match, not a spending cap the model may allocate under. A plan that runs 350 of 400 cases fails even though it is cheaper, because the policy is "the whole dev split", and the reason is statistical: at a pass rate near 50%, 400 cases give a 95% half-width of 1.96·√(0.25/400) = ±4.9 points, and a smaller, model-chosen subset would give a wider interval on a sample nobody vetted. A box labeled "check budget" that actually performs the exact-match comparison is mislabeled. Draw "check required runs" and "check budget" as two steps, because a test at a budget of exactly 1,620 cents (accepted) and one at 1,619 cents (rejected as budget_exceeded) show they fail independently.

Add a second requested eval set, tool-use-v1 with a 200-case dev split, and the required cost becomes 600 × 5 × $0.0081 = $24.30. Against $20.00, the full plan is rejected for budget_exceeded; the old one-set plan is rejected first for case_count_mismatch, because it no longer covers what was requested. A separately authorized $25.00 budget accepts the $24.30 plan. None of this launches anything. The status is accepted_draft, and the diagram must show launching and spending as later, separate boundaries.

Check the decision

What would falsify this diagram

The strongest test of an architecture diagram is to change one line of code and see whether the diagram silently becomes false. Change the budget comparison to read a field from the plan and the authority arrow moves. Remove the extra-field rejection and the schema contract changes, but the budget's source does not. Documentation drifts this way all the time, with no malice involved. It is also why a green local test only tells you that the validator held for the cases you wrote, in that run.

The next action: take a diagram you currently trust, pick its three most important boxes, and find the exact line where each reads its input. If you cannot find it, mark that part of the diagram unverified until someone does.

What an interviewer would ask

System-design conversations about agents often start at a whiteboard. A useful follow-up is "where does this value come from, in code?" for one arrow the candidate drew. A strong answer names the object, the loader and the check, and distinguishes a model-proposed value from an application-owned one. The same skill applies to reading other people's systems: in a research codebase, the fastest way to find why an eval number is wrong is often to trace which config the run actually read. Chapter 15 of the book builds the authority boundary this diagram depicts and measures what happens when untrusted text tries to cross it.