DM Explains · No. 040 · Voice
Why AI voice assistants finally stop sounding like a phone tree
The leap from stilted voice bots to natural conversation didn't come from bigger models — it came from rebuilding how they read time, in chunks as small as 160 milliseconds.
· 13 min read
Read me first
Voice assistants are not getting less stiff because the models underneath got bigger — they're getting less stiff because they stopped waiting for whole sentences before doing anything. Full-duplex interaction models process audio in slices as short as 160 to 200 milliseconds, which is why they can be interrupted mid-word and still make sense.
Papa calls support → 1 cascaded system waits full sentences, TTFB 4.9s → 2 streaming model processes 200ms chunks both ways → 3 block-causal attention lets it listen while talking, 160ms units → 4 Papa interrupts, Cerebellum aborts in <1ms, 100% resolution → 5 aware serving replays TTFB 1.55x faster, RTF 0.107 → 6 JAVIS-style joint synthesis lip-syncs the reply, ~200x lower initial latency than baseline
Papa calls customer support. He explains the problem in full, the bot waits, pauses, and answers — then he tries to correct himself and the bot talks straight over him, unaware anything changed. Everyone has had this call. The gap between that and a real conversation isn't intelligence, it's timing: a half-second of extra turn-taking latency, or an assistant that can't be interrupted, is the difference between something people tolerate and something people actually trust enough to talk to.
Everyday picture. When Papa tells a story on the phone, everyone waits silently until he finishes a full sentence before reacting — it feels stiff. But Bruno barks the instant the doorbell starts ringing, not waiting for it to finish, and Amma can keep stirring the pot while answering a question Arjun asks mid-sentence.
Where the picture breaks. Bruno's bark is a simple reflex with nothing behind it. The interaction models still run genuine language and audio generation within each tiny slice of time — far more demanding than a dog's bark, even if it happens just as fast.
Why it exists now
Through 2026, several labs pushed this idea from research papers into previews and production systems. Thinking Machines announced a research preview of multi-stream, full-duplex "interaction models," reporting a 0.40 second turn-taking latency on the FD-bench benchmark, against 0.57 seconds for Gemini-3.1-flash-live and 1.18 seconds for GPT-realtime-2.0, as reported by VentureBeat. Around the same stretch, a cluster of papers — Wan-Streamer, JAVIS, Gander, LiveServe — converged independently on the same design pattern: stop treating a conversation as a sequence of complete turns, and start treating it as a continuous stream sliced into fractions of a second.
What changed is not a smarter model but a different relationship with time. Cascaded systems treat an utterance as a single unit to be completed, converted, and answered. Streaming interaction models treat time itself as the thing to process incrementally.
How it works
Papa says "I need a refund" → cascaded: 1 waits for silence → 2 ASR→LLM→TTS queue sequentially, TTFB 4.9s → streaming: 3 160ms audio+video chunks interleaved → 4 Papa interrupts, barge-in resolves with <1ms abort latency, 100% resolution → 5 a visual cue mid-response routes to the cerebellum while the brain keeps reasoning → 6 TTS RTF 0.107 serves audio as it's generated → turn-taking latency 0.40s vs 1.18s for a cascaded baseline
Cascaded pipeline
Three stages, three waits, one stiff conversation
What it is. The old architecture: speech-to-text converts what was said into text, a language model generates a reply, and text-to-speech turns that reply into audio — three separate stages, run one after another.
Why it exists. It's modular. Each piece can be swapped, improved or outsourced independently, and it was the only practical way to build a voice assistant before models could handle raw audio end to end.
Where it breaks. Each stage has to finish before the next starts, and each handoff loses information — tone, hesitation, overlap — that never survives the trip through plain text. Even a well-engineered version of this pipeline, CrossOracle, reported a mean time-to-first-byte of 4.9 seconds across 8,000 conversational turns.
How that's handled. Some cascaded systems compensate with clever engineering rather than redesign — CrossOracle used a parallelised asynchronous text-to-speech queue to resolve barge-in instantly even while the rest of the pipeline stayed slow.
In technical terms
In technical terms: a cascaded voice pipeline runs automatic speech recognition, then a large language model, then text-to-speech sequentially, with no shared representation between stages, per the CrossOracle paper.
Chunked streaming
Listen and speak in 200-millisecond slices
What it is. Audio and video are cut into very short time slices and processed as they arrive, instead of waiting for a complete utterance.
Why it exists. Waiting for a full turn before doing anything is exactly what makes a system feel like a phone tree. Wan-Streamer supports streaming units as short as 160 milliseconds at 25 frames per second, and Thinking Machines' interaction models process 200-millisecond chunks of input and output simultaneously, per VentureBeat.
Where it breaks. Smaller chunks mean far more frequent computation — the system is doing work continuously rather than in bursts, which demands serving infrastructure built for that rhythm.
How that's handled. The chunk size is tuned as a trade-off, not minimised blindly: small enough to feel instant, large enough that the serving stack underneath can sustain it at scale.
In technical terms
In technical terms: Wan-Streamer processes interleaved audio-visual tokens in micro-turns of 160 milliseconds at 25 frames per second, reporting approximately 200 millisecond model-side response latency, per the Wan-Streamer paper.
Block-causal attn
Listening and talking at once
What it is. The model attends causally over blocks of interleaved tokens — speech it heard, speech it's generating — so it can keep "listening" to new input while it's still producing output.
Why it exists. This is the mechanism that actually enables barge-in and backchanneling: there's no need to stop a generation process and restart it, because listening and generating aren't separate modes.
Where it breaks. Overlapping speech streams, by design, don't produce one clean transcript — they produce two interleaved ones, which is harder to log, audit or review later than a strict turn-by-turn record.
How that's handled. Wan-Streamer builds this directly into a single Transformer that ingests and emits audio, visual and text tokens together, without separate external modules for language, speech or avatar generation.
In technical terms
In technical terms: block-causal attention lets the model attend only to prior blocks of tokens while continuing to emit new ones, supporting incremental streaming without a full restart, per the Wan-Streamer paper.
Fast reflex layer
A quick 'Cerebellum' for instant replies
What it is. A split between a fast component that handles real-time interaction and a slower component that handles complex reasoning.
Why it exists. Not every reply needs deep thought, and forcing every reply through a slow reasoning path is what makes a system feel laggy even when it's technically "thinking well." Gander names its fast component the Cerebellum and its slow one the Brain, coordinated through tool calling, per the Gander technical report.
Where it breaks. Tasks that genuinely need deep, multi-step reasoning are poorly served by the fast layer alone — asking the reflex component to handle something it's not built for just produces a confident, shallow answer quickly.
How that's handled. The two components don't share a queue: the Brain is invoked as a tool when the Cerebellum recognises a task is beyond its scope, rather than every request being routed through both.
In technical terms
In technical terms: Gander's Cerebellum-Brain framework coordinates a real-time interaction component and a complex-reasoning component via tool calling, per the Gander technical report.
Aware serving
Serve what's heard, not what's generated
What it is. The serving layer underneath the model — the infrastructure deciding what gets computed when — is made aware of playback progress, speech activity and interruptions, not just raw throughput.
Why it exists. A throughput-optimised server has no idea a user just interrupted; it keeps generating tokens nobody will ever hear. LiveServe exposes playback progress and barge-in events to its scheduler, prioritising sessions about to run out of audio and preloading cache while the user is still speaking, per the LiveServe paper.
Where it breaks. Without this kind of specialised serving layer, a deployment can't sustain many concurrent low-latency sessions — the model might be fast in isolation and still feel slow under real load.
How that's handled. LiveServe reported lowering P90 audio time-to-first-play by 1.55x on average, up to 2.21x, and improving completed-request throughput by 1.15x on average, up to 1.56x, compared with throughput-oriented serving.
In technical terms
In technical terms: LiveServe's scheduler uses playback-aware prioritisation and a KV-cache manager that preloads likely-needed state during user speech, built on vLLM-Omni, per the LiveServe paper.
Joint AV synthesis
One representation, voice and face together
What it is. Speech and facial motion generated from one shared representation, instead of two pipelines that each generate their own stream and then get stitched together.
Why it exists. Synchronising two independently generated streams — one for voice, one for a face — is a constant source of lag and drift. JAVIS instead produces a unified representation through what its authors call a "Joint Refined Head," feeding synchronised speech and face streams from the same source, per the JAVIS paper.
Where it breaks. This benefit is specific to systems that generate a visible face or avatar alongside speech; a plain voice assistant with no face has nothing to synchronise and gains nothing from this step.
How that's handled. JAVIS was trained on JAVIS-Syn, a 334-hour synthetic corpus of emotionally rich multi-turn and assistant-style audio-visual dialogue, and reported roughly 200 times lower initial latency than a strong baseline.
In technical terms
In technical terms: JAVIS generates speech and facial motion from a shared representation via a Joint Refined Head, reporting approximately 200x lower initial latency than its baseline, per the JAVIS paper.
Compared with what came before
| Cascaded pipeline | Streaming interaction model | |
|---|---|---|
| Processing unit | A complete utterance | A 160–200 millisecond chunk |
| Barge-in | Bolted on, or handled by a fast queue layer | Native to the model's attention design |
| Modules | Separate speech-to-text, language model, text-to-speech | One model, interleaved audio/visual/text tokens |
| Turn-taking latency | 1.18 seconds (GPT-realtime-2.0, per VentureBeat) | 0.40 seconds (TML-Interaction-Small, per VentureBeat) |
| Auditability | Clean, turn-based transcript | Overlapping streams, harder to log cleanly |
The honest judgement: a cascaded pipeline can be made fast in places — CrossOracle's abort latency for barge-in was under 1 millisecond even without any architectural rebuild — but speed in one stage doesn't erase latency built into the handoffs between the others. A unified streaming model removes the handoffs altogether, at the cost of a transcript that's genuinely harder to read back afterwards.
Where it breaks
Not a reasoning upgrade
Faster turn-taking is not the same as better thinking. A system tuned for sub-second responsiveness, like Gander's Cerebellum, is deliberately kept separate from the slower component meant for complex, multi-step reasoning — pushing a hard problem through the fast layer alone gets a quick, shallow answer.
Full-duplex response speed depends on infrastructure most teams underestimate. LiveServe's gains — up to 2.21x faster time-to-first-play, up to 1.56x more throughput — only show up when the serving layer itself is built to be interaction-aware; a generic throughput-optimised server won't deliver them even running the same underlying model. And the overlapping, interleaved transcript these systems produce is a genuine cost for anything needing a clean audit trail, a call-centre compliance log, or a simple record of who said what and when.
Numbers to carry
| Figure | Value |
|---|---|
| TML-Interaction-Small turn-taking latency (FD-bench, 2026) | 0.40 seconds |
| Gemini-3.1-flash-live turn-taking latency (2026) | 0.57 seconds |
| GPT-realtime-2.0 (minimal) turn-taking latency (2026) | 1.18 seconds |
| TML-Interaction-Small score on FD-bench V1.5 | 77.8 vs 46.8 for GPT-realtime-2.0 (minimal) |
| Wan-Streamer streaming unit size | 160 ms at 25 fps |
| Thinking Machines interaction model chunk size | 200 ms |
| Wan-Streamer model-side response latency | ~200 ms |
| CrossOracle mean time-to-first-byte (8,000 turns, 2026) | 4.9 seconds |
| CrossOracle barge-in abort latency | under 1 millisecond |
| CrossOracle mean TTS Real-Time Factor | 0.107 |
| LiveServe P90 time-to-first-play improvement | 1.55x average, up to 2.21x |
| LiveServe completed-request throughput improvement | 1.15x average, up to 1.56x |
| JAVIS initial latency vs strong baseline | ~200x lower |
| JAVIS-Syn training corpus size | 334 hours |
Myths worth killing
| Commonly repeated | What is actually true |
|---|---|
| "Faster voice AI means a bigger model" | The dossier's gains come from re-architecting how time is processed — chunking input into 160 to 200 millisecond slices — not from larger models. |
| "A fast cascaded pipeline is the same as a native multimodal one" | CrossOracle shows a cascaded pipeline can resolve barge-in in under 1 millisecond, yet still carry a 4.9 second mean time-to-first-byte; speed in one stage doesn't remove latency elsewhere. |
| "Multimodal means low-latency" | Accepting audio, video and text is not the same as being built for full-duplex interaction; many multimodal models still require a complete input before producing a complete output. |
| "This is just a better video call" | Interaction models address the model's own processing latency and turn-taking behaviour, not network transport — the network journey between a user and a server still adds its own delay on top of whatever the model itself achieves. |
Worked example
Papa's support call, traced end to end:
- Papa starts speaking: "I was charged twice for—" and pauses mid-sentence to correct himself.
- In a cascaded system, the pipeline waits for Papa to stop speaking entirely, converts the full utterance to text, generates a reply, and only then begins speaking — in CrossOracle's measured deployment, a mean 4.9 seconds passes before the first byte of a reply arrives.
- Papa tries to interrupt partway through the reply; in a fast cascaded system like CrossOracle, the abort is resolved in under 1 millisecond because the text-to-speech queue is built to be cut off instantly, even though the rest of the pipeline is slow.
- In a native interaction model, the assistant is already processing audio in 160–200 millisecond chunks as Papa speaks, so it starts generating a response within roughly 200 milliseconds of his pause, per Wan-Streamer's reported figures, rather than waiting for a full sentence boundary.
- The assistant backchannels — a short "mm-hm" — while Papa is still talking, made possible by block-causal attention letting it listen and generate at once rather than needing to stop and restart.
- Illustrative: if Papa's full call runs six exchanges, the gap between 1.18 seconds and 0.40 seconds per exchange adds up to roughly 4.7 seconds of pure waiting saved across the call — not a measured figure, just the shape of why it's noticeable.
- The serving layer, if built like LiveServe, has already started preloading cached context while Papa was speaking, so the reply doesn't stall waiting for infrastructure to catch up with the model.
Should you use it?
Full-duplex interaction models earn their complexity when a product genuinely needs natural spoken conversation with interruption and overlap — voice assistants, customer-support agents, anything where a half-second of lag is the difference between feeling heard and feeling stalled. They're a poor fit for tasks needing deep, slow, multi-step reasoning, where a fast reflex layer alone will just produce a quick shallow answer, and for anything needing a strict, auditable, single-stream transcript.
Say it in the room
"Use it when the product lives or dies on natural spoken interruption and sub-second response; skip it when the task needs deliberate multi-step reasoning or a clean auditable transcript more than it needs speed."
Lines to carry
- "Responsiveness isn't a bigger brain, it's a faster clock."
- "A conversation that waits for full sentences isn't a conversation."
- "Serve what's heard, not what's generated."
- "Reflexes and reasoning don't have to share a queue."
- "Fast in one stage doesn't erase latency built into the others."
FAQ
Is this just a bigger or newer language model under the hood?
No — the dossier's gains come specifically from processing audio and video in small chunks (160 to 200 milliseconds) and removing the handoffs between separate speech-to-text, reasoning and speech-generation stages, not from scaling up model size.
Can a cascaded pipeline ever feel this fast?
Parts of it can. CrossOracle resolved barge-in in under 1 millisecond using a parallelised text-to-speech queue, without rebuilding the whole pipeline. But its mean time to a first reply byte was still 4.9 seconds, because the earlier stages hadn't been redesigned.
Does this replace the need for good serving infrastructure?
No — it makes serving infrastructure more important, not less. LiveServe's reported gains, up to 2.21x faster time-to-first-play, only appear when the server itself is aware of playback and interruption events; running the same model on generic throughput-optimised serving won't deliver them.
Why does a face-generating system need a separate approach from a voice-only one?
Because keeping a generated face synchronised with generated speech is its own problem. JAVIS addresses that specifically by generating both from one shared representation, reporting roughly 200 times lower initial latency than a strong baseline — a benefit that has nothing to offer a system with no face to synchronise.
Is the overlapping speech harder to review later?
Yes, by design. A full-duplex system that lets both sides talk at once produces interleaved streams rather than a single clean transcript, which is genuinely harder to log and audit than strict turn-based systems.
Glossary
- Cascaded pipeline — a voice-AI system built from separate stages: speech-to-text, then a language model, then text-to-speech, run one after another.
- End-to-end / native multimodal model — a single model that directly processes raw audio, video and/or text and produces output in one of those forms without separate modules for each step.
- Full-duplex — both sides of a conversation can speak and be heard at the same time, rather than taking strict turns.
- Barge-in — when a user interrupts the AI while it's still speaking, and the system stops and responds to the new input.
- Backchanneling — short signals like "mm-hm" made while listening, to show engagement without taking over the conversation.
- Turn-taking latency — the time it takes for a system to start responding after the user's turn ends or is interrupted.
- Time-to-first-play / time-to-first-byte — the time between a request being made and the first piece of audio or data actually reaching the user.
- Block-causal attention — a way for a model to attend only to prior blocks of tokens while continuing to process new ones incrementally.
- KV cache — stored intermediate computations from earlier in a conversation that a model reuses instead of redoing the work from scratch.
- Real-Time Factor (RTF) — how long it takes to generate audio compared with the duration of that audio; lower means faster than real time.
- FD-bench — a benchmark built specifically to measure full-duplex, real-time conversational interaction quality.
Sources
- Thinking Machines shows off preview of near-realtime AI voice and video conversation with new interaction models — VentureBeat, 2026
- CrossOracle: An Agentic Framework for Real-Time Expert Personas with Parallelized Acoustic Synthesis — ACL Anthology / SIGDIAL, 2026
- LiveServe: Interaction-Aware Serving for Real-Time Omni-Modal LLMs — arXiv, 2026
- Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models — arXiv, 2026
- Omni Interaction Agent Technical Report (Gander) — arXiv, 2026
- JAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents — arXiv, 2026