Voice AI latency:
where the milliseconds go.

Ask anyone who has tried an early AI phone agent what felt wrong, and the answer is usually the pause. The words were fine; the timing wasn't. A gap of a second or more before every reply makes callers wonder whether the line has dropped, repeat themselves, or talk over the agent just as it starts speaking.

Latency is not one number. It is a stack of small delays, each added by a different part of the system. Understanding the stack is the only way to make it shorter.

How fast people take turns

Conversation is remarkably quick. In a review of turn-taking research, Stephen Levinson and Francisco Torreira describe the gaps between turns as short, "of the order of 200 ms", while the time it takes to plan and produce speech is much longer, over 600 ms. People manage this by predicting when the other person will finish and preparing their reply in advance.

A study by Tanya Stivers and colleagues, published in PNAS in 2009, compared turn-taking across ten languages from around the world. All of them showed the same tendency to avoid overlap and minimise silence between turns. Average gaps varied by language, but only within a narrow band around the cross-language average.

The implication for voice AI is stark. An agent that takes a full second to respond is around five times slower than a typical human turn. It does not need to match people exactly; callers are forgiving of a short pause for something that sounds like thought. But every few hundred milliseconds removed makes the conversation feel noticeably more natural.

Where the delay comes from

On a phone call to an AI agent, the delay between the caller finishing and hearing a reply is made up of:

1. The phone network. Audio travels from the caller's phone, through their operator, to the platform that hands the call to the AI. Mobile networks, international routes and internet transport all add time.
2. Buffering and transport. Audio is sent in small packets, typically every 20 milliseconds, and jitter buffers hold a little back to smooth out uneven arrival.
3. Deciding the caller has finished. This is often the biggest single delay. A system that waits for a fixed silence before replying adds that silence to every turn. Twilio notes that fixed end-of-turn defaults are commonly set around 500 ms.
4. Understanding and reasoning. In a cascaded system, transcription and then the language model's first output. In a speech-to-speech model, the model's own processing.
5. Generating speech. The time until the first audio of the reply is ready.
6. The journey back. The same network path in reverse.

A worked budget

Twilio has published latency targets for a cascaded voice agent. Its target for the full mouth-to-ear turn gap is about 1,115 ms, with an upper limit of about 1,400 ms. Inside that, it budgets roughly 350 ms for speech-to-text, 375 ms for the language model's first token and 100 ms for the text-to-speech engine's first byte, with the remainder going to the network and media handling. Twilio describes the public network leg as the slowest and least reliable part of the path.

Speech-to-speech models remove some of the hand-offs. OpenAI, for example, reports a turn-taking latency of 0.798 seconds for its GPT-Live 1 model, compared with 1.41 seconds for its previous realtime model. That is the vendor's own measurement, and like most vendor figures it describes the model, not the phone network around it.

The network sets a floor

Telephone engineers have studied delay for decades. ITU-T Recommendation G.114, widely cited in network design, treats one-way delays up to 150 ms as acceptable for most applications and delays above 400 ms as unacceptable for general network planning. Those figures are for a person-to-person call. An AI agent's processing comes on top of them, which is why a system that feels quick on office Wi-Fi in a browser can feel sluggish on an international mobile call.

Geography matters too. If the caller is in Dubai, the telephony platform's nearest media edge is in Europe and the model runs in the United States, the audio makes that journey twice per turn. Where you can choose regions, choose them deliberately and measure.

How to make it faster

• Use smarter turn detection. A model that predicts the end of a turn from the words and intonation can reply sooner than a fixed silence timer, and wait longer when the caller is mid-thought.
• Stream everything. Transcribe while the caller speaks, start generating as soon as possible, and play the first words of the reply while the rest is still being produced.
• Keep the voice model light. Let the voice keep talking and delegate slow lookups and reasoning to a backend, rather than blocking the conversation.
• Acknowledge slow operations. When a lookup genuinely takes time, a short natural phrase such as "let me check that" is better than silence.
• Avoid unnecessary audio conversion. Phone audio is 8 kHz G.711. Models that accept it directly avoid a resampling step.
• Put components close together. Minimise the distance between the telephony edge, your application and the model.
• Handle interruptions instantly. When the caller starts speaking, stop playback straight away. Talking over a caller feels worse than a pause.

Measure it properly

Measure the gap the caller actually experiences, from the end of their speech to the start of the agent's reply, on real phone calls from the places your callers ring from. Look at the median and at the slow tail, such as the 95th percentile: one painful pause in twenty calls is what people remember. Change one thing at a time and measure again, because improvements in one stage are easily hidden by noise in another.