The Latency Budget of a Voice Turn

A voice agent's speech recognizer is good, its synthesized voice sounds natural, and its model is fast. The call still feels slow. The usual first move is to swap in a faster model. This lesson shows why that rarely closes the gap, using two published measurements of human conversation and a budget you can check line by line.

The human numbers

Two research results set the target.

Put them side by side. Planning even one word takes about 600 ms, yet the typical gap is about 200 ms. The arithmetic only works if the listener starts planning before the speaker finishes: 600 − 208 = 392 ms of planning, at least, has to happen while the other person is still talking. Levinson and Torreira reach the same conclusion: in their account, planning a response begins as soon as the speech act of the incoming turn is clear, while that turn is still going.

A cascaded turn as a sum

A cascaded agent chains speech-to-text, a language model and text-to-speech. Measure the turn where the caller feels it: from the last sound of their speech to the first sound of the reply at their ear. Call that L.

L = d_up + t_eot + t_asr + t_llm + t_tts + d_down

d_up, d_down  network and telephony delay, caller to agent and back
t_eot         end-of-turn wait: how long the system waits before deciding the caller is done
t_asr         time to the final transcript after that decision
t_llm         time until the model has written the first phrase the synthesizer can speak
t_tts         time from that phrase to the first audio byte

View full-size diagram

Every box starts only when the one before it ends. That is what "reactive" means, and it's why the stages add.

The model term needs one more step. The book's chapter 9 derives the time to the first token as T_first = b·n + c·n² + d (prefill of an n-token prompt plus one decode step) and each later token as d more. A synthesizer usually needs a phrase, not one token, so if the first speakable phrase is k tokens long:

t_llm = T_first + (k − 1) · d

Plugging in numbers you can cite

Here is what can honestly be filled in today, and what you have to measure.

Now the arithmetic for the model term with a 12-token first phrase:

qwen2.5:0.5b   t_llm = 0.103 + 11/213 = 0.103 + 0.052 = 0.155 s
qwen2.5:1.5b   t_llm = 0.128 + 11/129 = 0.128 + 0.085 = 0.213 s

Those are small local models with short prompts. A hosted model adds its own network hop and queueing, and a long conversation history raises T_first through the b·n + c·n² prefill term. Measure your own with the book's chapter 9 experiment.

The network sits on both sides of any phone call, human or agent: each party hears the other one-way delay late, so the gap one caller perceives includes both legs. To sound like a person on the same line, the agent's own internal gap, from the caller's audio arriving to its reply leaving, should be close to the human gap. Take the Stivers mean, 208 ms, as that target. The smaller book model alone uses 155 of those 208 ms, which leaves 53 ms for end-of-turn detection, the final transcript and the first audio from the synthesizer, before any hosted-model overhead. The end-of-turn times Deepgram reports above are far larger than 53 ms on their own.

For scale on the whole turn: Hamming, a company that sells voice-agent testing, states that the industry median P50 is 1.5 to 1.7 seconds, from its own production telemetry (Hamming). The method isn't published, so treat it as a vendor's report, not a benchmark.

Why a faster model is not enough

A faster model shrinks one term of a sum that only starts after the caller stops. In the budget above, the end-of-turn wait can dominate, and it is not a compute cost at all: it's a decision under uncertainty about whether the caller is done. A model that was infinitely fast would still leave d_up + t_eot + t_asr + t_tts + d_down.

The fix humans use is overlap: start before the turn ends. Voice systems do the same thing with speculative execution. Deepgram's Flux, for example, can emit an early EagerEndOfTurn event so the agent starts the model call, then a TurnResumed event if the caller keeps talking, in which case the speculative call is canceled (Flux configuration). Deepgram reports that at eager thresholds of 0.3 to 0.5 the early event arrives 150 to 250 ms sooner, at the cost of 50 to 70% more model calls. That is a latency-for-compute trade you can price: if a fraction s of turns trigger a wasted speculative call, model spend per turn scales by 1 + s, so 1.5 to 1.7 times at those reported rates. The book's chapter 19 covers the cost side as tokens times price.

Tails and load

Two more facts you need before you measure anything.

First, the tail of a sum is not the sum of the tails. The p95 of L depends on how the stages vary together, so compute it from end-to-end measurements of the same turns, never by adding each stage's p95. Callers feel the tail: an average hides the turn that made someone hang up.

Second, stage times grow under load. By Little's law, the average number of requests in flight equals the arrival rate times the time each one takes (L = λ·W). With 20 turns a second each holding the model for 0.4 s, 8 requests are in flight on average; if your serving setup batches fewer than that without slowing down, turns queue and t_llm grows. The book's chapter 10 works through queues and utilization.