The Phone Agent Is Judged in Milliseconds, Not on Its Answers

Text-based assistants can take a couple of seconds to reply and nobody minds; the interface shows something is happening and the reader is not waiting on a live line. A voice agent answering a phone has no such cover. Human conversation turns over in a few hundred milliseconds, and a system that consistently takes two seconds to respond is experienced not as thoughtful but as broken, or as a recording. Callers talk over it, repeat themselves, and hang up. That single constraint reorders the whole design. Speech has to be transcribed as it arrives rather than after the caller stops; the system has to decide, from acoustics and content, whether the caller has finished or merely paused; the language model has to start producing a response before it has finished reasoning about it; and synthesis has to begin speaking the first words while the rest are still being generated. Every stage is streamed, and the budget for the entire chain — from the caller's last syllable to the agent's first — is measured against what a person would do. The consequence is that voice deployments favour smaller, faster models and tighter scopes than text ones, and put a great deal of engineering into what happens at the boundaries: how the agent handles being interrupted, how it signals it is still there during a lookup, and how quickly it hands off to a human when the conversation leaves the script. Accuracy still matters, of course, but it is bounded by the same clock; a perfect answer delivered after an awkward silence is scored as a failure by the only judge that counts. For businesses evaluating such systems, the practical advice is to test them the way callers will meet them — on a real line, from a mobile, with background noise, interrupting mid-sentence — and to look at the distribution of response times rather than the average. A demo that responds instantly in a quiet room and stalls on the second question in a car park has told you what you needed to know.