Week 0: Your Agent Passes in Text and Fails on a Real Call
Prerequisite: none. This is a short framing lesson. After this lesson, you can: say which properties of a voice agent a transcript cannot show, and name the three measurements this course teaches you to take: turn latency, barge-in behavior, and task success with an error bar.
This is an elective in the profrod.ai track for engineers heading into applied-AI roles. Voice is where agent engineering meets hard real-time limits: the agent has a latency budget measured in hundreds of milliseconds, it can be interrupted mid-sentence, and it can't take back what it already said. If you might build or evaluate a voice system on an applied-AI team, or you get a system-design question about one, this course gives you the vocabulary, the arithmetic and a measurement protocol you can run yourself.
What a transcript hides
If you've tested your agent by reading its transcripts, you've checked what it said. A transcript does not record:
- how long the caller waited after they stopped talking;
- whether the agent talked over the caller, or stopped when the caller said "mm-hmm";
- whether a pause meant "I'm done" or "I'm breathing";
- what the agent kept doing after the caller said "stop."
Each of those is a timing or concurrency property. Text testing is blind to all of them, because a chat has an explicit end of turn (the send button) and nothing runs between turns.
What you'll be able to do
Write the latency of one voice turn as a sum of stages (network, end-of-turn detection, speech recognition, the model's time to first token, speech synthesis) and compare it with cited measurements of human turn-taking.
Treat end-of-turn detection and interruption handling as decisions with two error types each, and build the cancellation path that makes "stop" actually stop.
Report task success with a confidence interval, compare two agent versions on the same scripted calls, and measure latency at the caller's ear rather than inside your own logs.
The six failures this course walks through are the places where a text-trained instinct gives the wrong answer on a call. They collapse into five patterns, and none of them waits on a better model.
Why an applied-AI team cares
A voice agent is an agent system with a clock attached. The skills transfer directly to lab work: decomposing a latency number into stages you can measure, reasoning about decisions under uncertainty, and reporting an evaluation result with its error bar instead of a single percentage. The book derives the pieces this course leans on: where a model's reply time goes in chapter 9, and how to put error bars on an evaluation in chapter 16.
Honesty note
The talk this course grew out of showed live demos on a commercial speech stack. Those demos and their numbers are not published, so this course does not quote them. Every number here is either arithmetic shown on the page, a measurement recorded in the book, or a vendor or research figure with its source linked and labeled. Where a number matters and nobody has published a clean measurement, lesson 11 gives you the protocol to take it yourself.
Who this is for
You should be comfortable with an LLM calling tools. No voice or telephony background is needed: speech recognition (ASR), speech synthesis (TTS) and SIP each come up where they cause a failure.
The human numbers from turn-taking research, the stages of a cascaded turn, and the arithmetic that shows why a faster model alone can't close the gap.
Reply here and it goes straight to Rod. Same as replying to one of his emails.
