Intermediate

Voice Agents, Not Chatbots: Latency, Barge-In, and Evaluation

A voice agent can read perfectly in a transcript and still fail a live call, because voice adds a clock, overlap and speech that can't be taken back. This elective treats those as engineering problems: a voice turn as a sum of stages set against measured human turn-taking, end-of-turn detection and barge-in as decisions with two error types each, and cancellation that reaches every running operation when a caller says 'stop'. You report task success with a confidence interval, compare two agent versions on the same scripted calls, and measure response gaps from a two-channel recording. Each number is arithmetic, a measurement recorded in the book, or a labeled citation.

MethodThis course teaches a discipline. The examples use today's tools; the method is meant to outlast them.

71 min read

12 lessons

Rod Rivera

Professor

Desk

First lesson open

Start the first lesson

Every lesson is free. Lessons marked open need no account; the rest open once you register.

AI agentsvoice agentslatencyagent evaluation

What You'll Learn

Write the latency of a cascaded voice turn as a sum of stages and fill each one from a cited source, a book receipt, or your own measurement
Explain with arithmetic, using published turn-taking research, why a reactive pipeline can't match human timing however fast its model is
Compare cascaded and speech-to-speech systems by what each lets you inspect and evaluate
Treat end-of-turn detection as a decision with two error types, and add a completeness layer with a hold ceiling above the recognizer
Build a two-tier classifier that separates a barge-in from a backchannel, and measure its false and missed barge-in rates and its stop latency
Design cooperative cancellation that reaches the model stream, tools, synthesizer and audio queue, and handle side effects that already ran
Use deterministic gates for call-ending, disclosure and transfers, and state as the only source of spoken facts
Report task success with a Wilson interval, size a test set, and compare two agent versions with a paired test
Run a measurement protocol that records both sides of a call and reports response gaps, stop latency and error rates with intervals

Prerequisites

Comfort with an LLM calling tools; this course doesn't re-teach tool-calling mechanics
Enough Python to read and run short listings on Python 3.12 (Colab works)
Helpful, not required: chapters 9 (prefill, decode and latency) and 16 (evaluation with error bars) of the book
No prior voice or telephony experience; ASR, TTS and SIP are introduced where they cause a failure