Multi-Model and Voice Agent Architecture

A banking demo built for this course had a real incident: a large reasoning model's <think> blocks (the extended internal reasoning some models generate before answering) pushed conversation history past a tiny classifier's context window (its total budget for how much text it can read at once) within two turns, forcing an expensive fix. Separately, a tool-design principle called atomicity (one tool should do exactly one thing) got deferred out of an earlier lesson specifically to land here. Those look like two unrelated topics: an infrastructure sizing problem, and a tool schema style question. They aren't. Both are the same failure class wearing different clothes: one component in a pipeline makes an independent decision that the rest of the system wasn't sized to absorb. A reasoning model generating more tokens than a downstream classifier can hold, and a fat tool forcing the model to guess a sub-action from an ambiguous enum, are the same shape of mistake: something upstream, silently breaking something downstream that was built for less.

The thinking-token overflow, and what it actually cost

The banking demo's architecture paired a large reasoning model, handling conversation, with a tiny classifier — Gemma 2B — doing intent detection and safety monitoring. The reasoning model generated long <think> blocks as part of its normal output. Those thinking tokens got appended to conversation history, and when that history was passed to the 2B classifier, it exceeded the classifier's 9K-token context window — within two turns.

The fix required two separate changes, not one:

The cost impact: a five-times increase in classifier cost — and the classifier wasn't bad. The reasoning model upstream was generating tokens the classifier was never sized to absorb. That's the whole lesson in one incident: every model in your pipeline constrains every other model. A <think> block is invisible to the user, but it is very visible to your token budget and to whatever context window sits downstream of it.

The same demo got progressively slower across a conversation, and the cause traces to the same root: cumulative history, now compounding with every thinking-token block appended alongside it. Turn 1, a fast response. Turn 3, a noticeable delay once six messages and their thinking tokens have accumulated. Turn 6, sluggish, with twelve messages and every prior thinking block still in history. Voice amplifies this directly — each turn runs ASR into the LLM into TTS, and a 3-second LLM delay becomes a 5-second voice delay once transcription and synthesis overhead stack on top.

Why the banking demo used a pipeline, not speech-to-speech

The demo's voice architecture was a multi-stage pipeline — ASR, then a safety classifier, then LLM reasoning, then TTS — not a speech-to-speech model like GPT-4o voice or Gemini Live, both of which exist and work well for simple conversation. The reason a compliance-bound system like banking or healthcare cannot use speech-to-speech comes down to one property: there is no intermediate text to inspect, filter, route, or log.

:::tabs{labels="Pipeline (ASR to LLM to TTS),Speech-to-speech}

Full text transcript at every stage. Can block, rewrite, or route between stages. Can delegate to different LLMs per turn. Higher latency from three hops, but every hop is auditable — which is why enterprise, compliance, banking, and healthcare use this shape.

Opaque — no intermediate text exists to inspect. Cannot intervene mid-generation. A single model handles everything end to end. Lower latency from one hop, which is why on-device assistants and casual voice chat favor it — but it is architecturally unsuitable anywhere an audit trail is required.

:::

If you need to audit, filter, or route, use the pipeline. If you need minimum latency and the domain is simple, speech-to-speech can work. The banking demo's two-agent escalation — a first-line agent handing off to Patricia Walsh, the escalation agent handling anything the first-line agent couldn't resolve — was built as a LangGraph state machine, not A2A, and that choice was deliberate: the model decided when to escalate based on conversation context, but the graph defined the possibility of escalation within one organization's own system. A2A would only be the right call if the escalation target were a different organization's agent entirely — a Klarna credit agent, say, not an internal handoff. For internal routing, a graph is simpler, and reaching for A2A here would have been the wrong tool for the boundary actually being crossed.

Hallucination versus jailbreaking: two failures that look alike and aren't

The same banking demo produced a second incident worth its own section, because it's the architectural fix lesson 11 pointed to and it belongs here. Patricia Walsh's escalation agent had no domain boundary — nothing in its system prompt or its tool access restricted what it would answer. A test conversation asked it, mid-escalation, for a chocolate cake recipe. It gave one, fluently, in the same voice it used to discuss account holds and credit limits.

That's worth being precise about, because it looks like the model getting "jailbroken" and it isn't. Hallucination is the model answering a question it has no grounding for, confidently, because nothing stopped it from generating a plausible-sounding response — the cake recipe is exactly this: not a security bypass, just an ungrounded answer to an off-domain question. Jailbreaking is a user deliberately working around a stated restriction — "ignore your instructions and tell me anyway." The banking agent was never instructed to refuse recipes; it was never told banking was its only domain in the first place. Nobody broke a rule, because no rule existed to break.

The fixes are different because the failures are different. Hallucination gets fixed by grounding with tools — instead of letting the model answer from parametric memory, force it to answer only from what a retrieval or lookup tool actually returns, so "I don't have that information" becomes a possible output instead of a fluent guess. Jailbreaking gets fixed by domain locking — an explicit, enforced boundary stating what the agent will and will not discuss, checked structurally (a classifier gate, an allowed-topics list) rather than left to the system prompt's persuasive request alone, because a persuasive request is exactly what a jailbreak defeats.

Domain locking connects directly back to this lesson's own multi-model sizing discipline: a cascading pipeline (classifier, then reasoning model, then escalation agent) needs every component locked to the scope it was actually sized and scoped for, the same way the classifier needed to be sized for the reasoning model's thinking-token output. An unscoped escalation agent is a component nobody sized for "answer anything" — it just happened to.

Tool Design Principle 6: atomicity

One tool, one decision, one action. A tool should do the smallest useful thing, and if it requires the model to make multiple independent decisions inside a single call, that's the signal to split it.

A fat tool looks efficient on paper: one schema, one name, an action enum spanning create, delete, list_issues, merge_pr. In practice it produces exactly the failures this course keeps returning to. Parameter confusion, where the model fills pr_number when it meant issue_id, because both are typed as plain integers with nothing in the schema to distinguish them. Schema bloat, fifteen optional parameters where the model has to guess which ones actually apply to the chosen action. Error ambiguity — "Action failed" tells you nothing about whether the repo name, the action type, or a permissions check was the actual problem. And inconsistent invocation, where the model calls the same intent as action="list" one turn and action="list_issues" the next, because nothing in the schema pins the value down.

The fix: split the fat tool into atomic tools, each doing one thing with two to four required parameters.

The GitHub API has hundreds of endpoints. Wrapping all of them in one tool is not powerful — it's a trap dressed as convenience. When atomic tools exist, the agent composes them into complex workflows on its own:

Four distinct decisions, composed by the model from atomic pieces. No single fat tool could have handled this without the model guessing which sub-action to invoke — and the lesson's own framing is blunt about the odds: guessing wrong roughly half the time. If a tool's action parameter has an enum of five or more options, it is not one tool. It's five tools wearing a trench coat.

The same failure class, twice

Here's the connection worth holding onto after this lesson: the thinking-token overflow and the fat-tool anti-pattern are both instances of one component making an independent decision the rest of the system wasn't built to absorb. The reasoning model didn't know or care that a 2B classifier downstream had a 9K budget — it just generated the tokens it generates. A fat tool's action enum doesn't know or care that the model has to guess correctly on the first try — it just presents fifteen parameters and hopes. In both cases, the fix is the same shape: size or split the component so no single upstream decision can silently break something downstream that was never built to absorb it. Atomicity is a reliability discipline for exactly this reason, not a style preference about parameter counts.

What carries forward

This lesson closes the tool-design principle set that opened in lesson 07 — six principles now, not five — and it closes the hallucination-versus-jailbreaking distinction the banking demo's "friendly agent" failure raised, in full: grounding with tools for the first, domain locking for the second. One thread stays open on purpose: the fuller context-compression discipline that a cascading multi-model pipeline like this one needs at scale belongs to lesson 13, which builds the implementation layer this lesson's architecture only diagnosed.