Every AI voice agent has to do three things: understand what the caller said, decide what to say, and say it. For most of the last decade those were three separate systems. Since 2024, models that do all three in one step have become practical for real-time use. Both approaches are now in production, and the choice between them shapes how your agent sounds, how fast it replies, which languages it handles well and how much control you have.
The cascaded pipeline
A cascaded, or "chained", agent passes each turn through three components:
- Speech-to-text (STT) transcribes the caller's audio as it arrives and decides, with a turn-detection model or a silence timer, when they have finished.
- A language model (LLM) reads the transcript and the conversation so far, calls any tools it needs, and writes a reply as text.
- Text-to-speech (TTS) turns the reply into audio, streaming the first words while the rest is still being generated.
Platforms such as Twilio ConversationRelay, and frameworks such as LiveKit Agents and Pipecat, are built around this pattern and let you swap each component.
Strengths. You can choose the best component for each job: a speech recogniser that documents Gulf Arabic, a reasoning model you trust with tools, and a voice your customers like. Every turn passes through text, which makes it easy to log, filter, redact and test. You can mix vendors and change one piece without touching the others.
Weaknesses. Each hand-off adds delay. Twilio's published latency guide for cascaded agents sets targets of around 350 ms for transcription, 375 ms for the language model's first token and 100 ms for the voice's first audio, adding up to a mouth-to-ear target of about 1.1 seconds once the network is included. Text also discards information: the caller's tone, hesitation and emphasis are flattened into words, and the voice has to guess the right delivery from text alone. Interruptions need careful engineering, because each component has to be stopped and reset.
Speech-to-speech
A speech-to-speech model takes audio in and produces audio out. Under the hood it may still produce transcripts, but listening, reasoning and speaking happen in one model, often full duplex: it can listen while it talks. OpenAI's GPT-Live, which Clodia uses, Google's Gemini Live models and Amazon's Nova Sonic are examples.
Strengths. Fewer hand-offs mean quicker replies. Hearing the audio directly lets the model respond to tone, pick up on hesitation and deliver its answer with natural prosody. Interruptions and backchannels such as "mm-hm" are handled inside the model rather than stitched together.
Weaknesses. You get the vendor's voices and the vendor's language coverage. If the model is weak in a dialect your callers use, you cannot swap in a better recogniser. Tool use from inside a voice model can be less controllable, and long reasoning blocks the conversation. Debugging is harder because there is no clean text boundary between understanding and speaking.
Side by side
| Cascaded | Speech-to-speech | |
|---|---|---|
| Typical response time | Sum of three stages plus network | Usually shorter; one model |
| Interruptions | Must be engineered across components | Handled inside the model |
| Tone and prosody | Lost at the text boundary | Heard and produced natively |
| Language and dialect choice | Pick the best recogniser and voice | Limited to the model's coverage |
| Voices | Any TTS vendor, custom voices | The vendor's voices |
| Control and logging | Text at every step | Transcripts, but less separation |
| Vendor lock-in | Low; swap components | Higher |
The hybrid most systems end up with
The newest speech-to-speech designs separate talking from thinking. The voice model keeps the conversation flowing and hands anything that needs facts, reasoning or actions to a separate backend model with tools. OpenAI describes GPT-Live this way: a full-duplex voice that delegates deeper reasoning and actions to a backend model or agent.
Clodia is built on this pattern, and goes one step further: the voice model is given no tools at all. It delegates to a backend model that holds the business's playbook and a short list of tools, and those tools run on our servers, scoped to one business. The caller gets the natural feel of speech-to-speech; the business gets the control of a text-based backend. A caller cannot talk the voice into an action, because the voice cannot take any. Read inside a Clodia call for the details.
Which should you choose?
- Mostly English callers, conversation quality matters most: speech-to-speech, with a delegated backend for tools.
- Callers in a dialect your preferred speech-to-speech model handles poorly: a cascade with a recogniser and voice that document that dialect, or a speech-to-speech model you have tested carefully with native speakers.
- Strict compliance needs, such as redacting card numbers before anything is stored: a cascade, or a hybrid where you control the text layer.
- A custom brand voice: a cascade with your chosen TTS vendor, unless your speech-to-speech provider supports custom voices.
Whatever you choose, keep the architecture behind an interface. Both approaches are improving quickly, and the right answer for your callers may change within a year. Our model guide lists what is available today.
Sources and further reading
Hear Clodia for yourself.
Try a live conversation in your browser, or tell us about your front desk.