DM Explains · No. 039 · Architecture
Multi-modal coordination: when should an AI listen and when should it speak
Making a model see, hear and read is easy now. Deciding, instant by instant, whether to listen or speak inside one live conversation is the actual engineering problem.
· 16 min read
Read me first
Making a model see, hear and read is the easy part now; deciding, instant by instant, whether to listen or to speak inside one live conversation is the engineering problem nobody has fully solved. It genuinely gets better when the decision lives inside the model itself, but only when the model was built for that from the start.
A duplex support call: 1 audio chunked as caller speaks mid-sentence → 2 speech and reply tokens share one sequence → 3 model predicts listen while caller says 'mm-hmm' → 4 Cerebellum keeps the call alive with a quick reflex → 5 a real lookup question triggers a quiet Brain handoff → 6 the Brain backend could be swapped for a stronger reasoner without retraining → 7 a graph wires mic, screen and client events together → caller gets a grounded, timely answer
Amma is cooking while Dadi talks from the next room, Bruno barks at the door and the radio plays — she doesn't finish one sound before reacting to the next, she stirs, listens, answers and glances all in the same breath. Nobody designed that for her; it just happens.
Most voice assistants cannot do it. They wait for silence, then process, then speak, then wait again. That feels fine in a demo where one person asks one clean question. It feels infuriating in a real conversation, where someone interrupts mid-sentence, says "mm-hmm" while the assistant is still talking, or asks a follow-up before the first answer has finished. Gartner expects 80% of enterprise software to be multimodal by 2030, up from less than 10% in 2024 (Gartner, 2025), which means this gap between demo and real conversation is about to show up in a lot more products than it does today.
Everyday picture. Amma stirs the pot while Dadi talks from the next room, Bruno barks at the door and the radio plays — she answers Dadi, glances at Bruno and keeps stirring, all without finishing one sound before starting the next.
Where the picture breaks. Amma's mind doesn't split into a fast reflex module and a separate slow-thinking module that she calls via a formal handoff. Her coordination and her reasoning aren't architecturally separated pieces; in the systems below, they are two different components that explicitly pass structured context between each other.
Why it exists now
The problem is older than the tools used to solve it. For years, "multimodal" meant a model that could accept an image and some text in one prompt, process it once, and reply. That is reading a photo, not holding a conversation. It says nothing about what happens when audio keeps arriving while the model is still mid-reply, or when the user starts talking over it.
The usual patch was a pipeline: a voice-activity-detection (VAD) module decides when someone has stopped talking, hands a clean audio clip to a speech-to-text system, hands the text to a language model, gets a reply, and hands that to a text-to-speech system. Every arrow in that chain is a seam, and seams are where interruptions, backchannel noises like "uh-huh", overlapping speech and background noise go wrong. A 2026 technical report on a model called Gander argues exactly this: reliably coordinating spontaneous interruptions, agent-initiated engagement, background noise and backchannel acknowledgements is challenging for pipelines that rely on separately designed modules such as Silero VAD (arXiv, 2026).
What changed is that a model can now be built so the listen-or-speak decision sits inside it, not beside it. Gander, built on MiniCPM-o 4.5, is described as a native multimodal duplex interaction model with an asynchronous agent loop, meaning it was trained from the start to process streaming audio-visual input and its own generated text together, rather than having a decision module bolted on afterwards (arXiv, 2026). Separately, Gartner named Speech Reasoning Models a top strategic technology trend for 2027, aimed squarely at automating voice — the enterprise's highest-volume and least-automated interaction channel (Gartner, 2026). The pressure and the architecture arrived at roughly the same time.
How it works
Stream chunking
Nothing waits for a full sentence to begin
What it is. Incoming audio, video and text are sliced into small, time-aligned pieces instead of being buffered until a full utterance or a full turn finishes.
Why it exists. Without aligned chunks, the system has no way to reason about what is happening "now" across modalities — it has to wait for a pause before it even starts.
Where it breaks. If the chunks are too coarse, the system behaves like a turn-based pipeline wearing a streaming costume: it still waits, just in smaller increments.
How that's handled. Chunk size becomes a deliberate design parameter, tuned against the kinds of interruptions the system actually needs to catch.
In technical terms
In technical terms: Gander partitions streaming audio-visual inputs and the model's own generated text stream into temporally aligned chunks, which are then organised into a unified autoregressive sequence (arXiv, 2026).
One sequence
Perception and generation share a single timeline
What it is. The chunks of what the model hears and sees, and the chunks of what the model itself is saying, are folded into one sequence the model works through step by step.
Why it exists. Timing — who should be acting right now — gets learned jointly inside the model, instead of being managed by an external switch sitting between separate perception and generation systems.
Where it breaks. This requires a model purpose-built for it. Two separate off-the-shelf models, one for hearing and one for talking, cannot be stitched into one sequence after the fact; the joint timing has to be trained in from the start.
How that's handled. Gander is adapted from MiniCPM-o 4.5 specifically for this, rather than assembled from independent components (arXiv, 2026).
In technical terms
In technical terms: the model is organised so that audio-visual chunks and its own generated text stream share one autoregressive sequence, letting a single set of weights govern both what it perceives and what it produces (arXiv, 2026).
Listen or speak?
The decision lives inside the model, not beside it
What it is. For each chunk, the model explicitly predicts one of two states: listen now, or speak now.
Why it exists. It replaces the external VAD module that decides, from outside the model, when someone has finished talking — a decision that struggles with interruptions, overlapping speech and small backchannel noises.
Where it breaks. The prediction is only as good as the conversational data it was trained on. A model trained mostly on clean, polite, one-speaker-at-a-time dialogue will mispredict in noisy, multi-party, interruption-heavy settings.
How that's handled. The Gander report frames this explicitly as a response to VAD's limitations with dynamic conversational behaviours like spontaneous interruptions and background noise (arXiv, 2026).
In technical terms
In technical terms: within each temporally aligned chunk, the model predicts an interaction state — listen or speak — giving it intrinsic control over turn-taking rather than delegating that decision to an external voice-activity-detection module (arXiv, 2026).
Cerebellum
Fast reflexes keep the conversation alive
What it is. A lighter-weight component, named the Cerebellum, continuously tracks the live interaction — what is being said, seen and heard right now — and produces immediate conversational responses.
Why it exists. Casual conversation cannot wait for deep reasoning. If every "how's it going" triggered a slow, careful multi-step reasoning process, the conversation would stall constantly.
Where it breaks. Kept too simple, the Cerebellum gives fast but shallow answers to questions that actually needed real reasoning, and the conversation sounds glib rather than careless-but-charming.
How that's handled. The Cerebellum doesn't try to answer everything itself — it recognises when a question is beyond it and reaches for help.
In technical terms
In technical terms: Gander's Cerebellum handles realtime multimodal interaction, continuous perception and responsive conversational control, forming the fast half of a decoupled "Cerebellum-Brain" architecture (arXiv, 2026).
Illustrative: caller interrupts mid-sentence → 1 chunk aligns audio+text → 2 model predicts 'listen' on the interruption, then 'speak' after 'mm-hmm' → 3 Cerebellum keeps the call alive with a quick acknowledgement → 4 a real lookup question triggers a tool call with transcript and frames → 5 the Brain reasons over the account history → 6 synthesis merges the answer back into speech → caller gets a timely, correct reply
Brain handoff
Hard questions get asked quietly, in the background
What it is. When a question needs real planning, lookup or multi-step reasoning, the Cerebellum calls a separate, stronger component, the Brain, as a tool, without freezing the live conversation.
Why it exists. It resolves the tension between staying responsive and reasoning properly. One module cannot do both well at once, so the system splits the work and keeps talking while the Brain works.
Where it breaks. If the handoff is slow, or the Brain's answer lands oddly out of step with what's happening live, the seam between fast and slow shows through — the system suddenly sounds like it's reading from a different conversation.
How that's handled. The Cerebellum keeps the context flowing to the user while the Brain reasons asynchronously in the background, then folds the result back in once ready.
In technical terms
In technical terms: the Cerebellum invokes the Brain through tool calls, passing accumulated textual context, speech transcriptions and visual frames; the Brain reasons asynchronously and returns a result the Cerebellum incorporates (arXiv, 2026).
Swappable Brain
Upgrade the thinker without retraining the reflexes
What it is. The Brain component is designed to be replaced with a different, stronger reasoning model later, without retraining the Cerebellum or the rest of the interaction loop.
Why it exists. Reasoning models keep improving. Rebuilding the entire duplex system every time a better reasoning model appears would be wasteful.
Where it breaks. "Plug-and-play" assumes compatible interfaces. A radically different reasoning model may still need adapter work in practice, even if no full retraining is required.
How that's handled. The report describes the Brain as requiring no additional training within Gander, which keeps the swap closer to a configuration change than a rebuild.
In technical terms
In technical terms: Gander's Brain is a plug-and-play component requiring no additional training within the system, allowing different or stronger reasoning backends to be incorporated without retraining the core interaction model (arXiv, 2026).
Graph orchestration
Someone still has to wire the microphones to the model
What it is. Outside the model itself, infrastructure platforms treat a live multimodal application as a graph of media streams and events — publishing camera, microphone and screen streams, listening for model events, and sending client-side events back.
Why it exists. Not every model is natively duplex. Developers still need a way to wire multiple live media sources into whatever model they are using, including ones that were never built for this.
Where it breaks. This is plumbing, not coordination. It lets non-duplex models like Qwen3 VL or Gemini be wired into a live application, but the in-model listen-or-speak decision described above is a different thing entirely, and graph orchestration doesn't give you that.
How that's handled. Teams treat this layer as what it is: operational wiring around a model, useful on its own, and more useful again once the model underneath it is duplex-native.
In technical terms
In technical terms: Gabber's backend publishes camera, mic and screen streams, listens for graph events, and sends client events like button clicks or text input, letting modern multimodal models that process audio and video in addition to text and images, such as Qwen3 VL and Gemini, be orchestrated into live applications (Gabber, 2026).
Compared with what came before
| Pipeline with external VAD | Native duplex coordination | |
|---|---|---|
| Who decides to listen or speak | A separate voice-activity-detection module, outside the model | The model itself, per chunk, as part of generation |
| Handling interruptions | Struggles; treats talking-over as noise | Trained explicitly as part of interaction state |
| Backchannel noises ("uh-huh") | Often misread as a turn-ending signal | Modelled as a distinct, non-turn-taking event |
| Fast reply vs deep reasoning | One model tries to do both, or reasoning blocks the conversation | Split: Cerebellum replies fast, Brain reasons asynchronously |
| Upgrading the reasoning component | Usually means retraining or replacing the whole pipeline | Brain is designed to be swapped without retraining the rest |
| Build effort | Assembled from existing speech-to-text, LLM and text-to-speech components | Requires a model purpose-built for joint streaming |
The pipeline is cheaper to assemble because every piece already exists off the shelf. The duplex model is more coherent in conversation because the decision about who should be talking right now was trained in rather than bolted on — but it demands a model built for that from the ground up, which is a smaller menu of options than "any LLM plus any VAD".
Where it breaks
Native duplex coordination helps mainly when the conversation is genuinely live and messy — interruptions, overlapping speech, background noise. It helps less, and costs more to build, when the interaction is actually a clean sequence of one question and one answer.
Trained data, not understood conversation
The model predicting "listen" or "speak" per chunk is pattern prediction over training data, not a judgement about social cues. A system trained mostly on tidy, one-speaker-at-a-time dialogue will mispredict in noisy, multi-party settings it never saw enough of during training.
The Brain handoff is a seam, even when hidden
Passing accumulated text, transcriptions and visual frames to a separate reasoning module and waiting for an asynchronous result is still a round trip. If that round trip is slow, the live conversation either stalls or the eventual answer lands oddly out of step with what's already been said.
Graph-based orchestration infrastructure, of the kind Gabber describes, is not a workaround for either of these limits — it solves a different problem, which is wiring live media streams into a model's input and output, whether or not that model has any native listen-or-speak prediction at all (Gabber, 2026).
Numbers to carry
| Figure | What it means |
|---|---|
| 80% by 2030, up from less than 10% in 2024 | Gartner's forecast for the share of enterprise software and applications that will be multimodal (Gartner, 2025) |
| 40% by 2027 | Gartner's earlier prediction for the share of generative AI solutions that would be multimodal, made in September 2024 (Gartner, 2024) |
| Speech Reasoning Models, 2027 | Named by Gartner as a top strategic technology trend for 2027, targeting voice as the enterprise's highest-volume, least-automated channel (Gartner, 2026) |
| Gander, built on MiniCPM-o 4.5 | The 2026 technical report describing a native duplex interaction model with an explicit listen/speak prediction per chunk (arXiv, 2026) |
Myths worth killing
| Commonly repeated | What is actually true |
|---|---|
| "A model that accepts images and audio is already doing multimodal coordination." | Accepting several input types in one static prompt is not the same as continuously deciding, chunk by chunk, whether to listen or speak in a live stream. |
| "Voice-activity-detection is a solved problem." | The Gander report specifically names interruptions, backchannel noise and background noise as cases where external VAD modules struggle. |
| "You need one giant model to handle both fast chat and deep reasoning." | The Cerebellum-Brain split exists precisely because one component doing both tends to be either too slow for chat or too shallow for reasoning. |
| "Swapping in a better reasoning model means retraining the whole system." | The Brain is described as a plug-and-play component that does not require additional training within the system — though compatible interfaces still matter in practice. |
| "Graph-based media orchestration gives you duplex coordination." | It wires live streams into a model's input and output; it does not give a non-duplex model an in-model listen-or-speak decision. |
Worked example
A caller rings a duplex customer-support voice agent.
- The caller starts speaking. The system chunks the incoming audio in small, time-aligned pieces rather than waiting for the caller to finish a sentence.
- Midway through the agent's first reply, the caller interrupts with a new question. Because the model predicts listen-or-speak per chunk rather than relying on an external silence detector, it registers the interruption as a state change rather than as noise to ignore.
- While the agent is still mid-reply, the caller says "mm-hmm". The model's per-chunk prediction treats this as a backchannel acknowledgement, not a request to stop talking.
- The caller then asks a question that needs an actual account lookup — something the fast-responding Cerebellum cannot answer from context alone. The Cerebellum calls the Brain as a tool, passing along the accumulated transcript and conversational context.
- The Brain reasons asynchronously in the background. Illustrative only: if that lookup takes a few seconds, the Cerebellum keeps the conversation alive with a brief acknowledgement rather than going silent.
- The Brain returns its result, and the Cerebellum folds it into the ongoing conversation, continuing in the same voice rather than switching tone abruptly.
- If the support team later adopts a stronger reasoning model, the Brain can, in principle, be swapped without retraining the Cerebellum or the duplex listen/speak prediction underneath it.
Should you use it?
Native duplex coordination earns its complexity when a product genuinely runs a live, continuous, two-way conversation where interruptions and backchannel noise are normal — voice support lines, collaborative voice tools, anything where a caller talks over the assistant regularly. It is not worth the build when the interaction is really a single question followed by a single answer, or when latency tolerance is loose enough that a traditional request-response pipeline already feels fine.
The cheapest first experiment is not building a duplex model from scratch — it's measuring how often your current pipeline mishandles interruptions and backchannel noise in real transcripts, before deciding whether the fix needs to live inside the model or can stay as orchestration around it.
Say it in the room
"Use native duplex coordination when the conversation is genuinely live and people talk over each other; stick with a request-response pipeline when the interaction is really one question and one answer."
Lines to carry
- "Seeing and hearing are easy; deciding when to speak is the hard part."
- "A pipeline waits for silence; a duplex model never stops listening."
- "Fast reflexes keep the conversation alive; a slow brain keeps it right."
- "Plug in a stronger thinker without retraining the reflexes."
- "Graph orchestration is plumbing, not coordination."
FAQ
Is this just a better voice-activity-detection algorithm?
No. VAD is an external module that tries to guess when someone has stopped talking, based mostly on audio silence. The listen-or-speak prediction described here sits inside the model's own processing, chunk by chunk, learned jointly with everything else the model does.
Does the Cerebellum-Brain split mean two separate models running at once?
Functionally, yes — a lighter, fast-responding component handling live interaction, and a separate, stronger component invoked through tool calls when deeper reasoning is needed. The Gander report describes the Brain as requiring no additional training within the system, which is what makes the split practical rather than just theoretical.
Can I get this behaviour by wiring existing models together with infrastructure like Gabber's graph-based backend?
You can wire camera, microphone and screen streams into a model and manage events between them, which is genuinely useful. But that orchestration layer doesn't give a model an in-model listen-or-speak decision — it moves streams around a model that may still process them in a turn-based way underneath.
Why does this matter for enterprise software specifically?
Because Gartner expects the share of multimodal enterprise software to rise sharply — 80% by 2030, up from under 10% in 2024 — and separately named Speech Reasoning Models a top strategic trend for 2027, aimed at voice as the highest-volume, least-automated interaction channel. Coordination, not just multimodal input, becomes the bottleneck at that scale.
What's the realistic failure mode if a team gets this wrong?
Mispredicting listen-or-speak in noisy or multi-party settings, because the model wasn't trained on enough of that kind of mess — or a Brain handoff slow enough that the live conversation stalls or the eventual answer arrives oddly out of step with what's already been said.
Glossary
- Multi-modal coordination — architecture that lets an AI system process and act on speech, vision and text together in real time, rather than handling each modality separately.
- Duplex interaction — a mode where a system can listen and speak, or process input and generate output, concurrently rather than strictly taking turns.
- Turn-based interaction — the conventional pattern where a user gives an instruction and the system replies in a discrete cycle, one party at a time.
- Voice Activity Detection (VAD) — a technique or module that detects when someone is speaking, traditionally used to decide when a voice assistant should start or stop listening.
- Autoregressive sequence — a sequence a model generates step by step, where each new piece depends on everything observed or generated before it.
- Cerebellum-Brain architecture — a design splitting a system into a fast, responsive component for live interaction and a slower, more powerful component for deep reasoning and planning.
- Tool call — a structured request a model makes to invoke another component, model or function and receive a result back.
- Backchannel behaviour — small verbal cues like "uh-huh" that a listener gives during conversation to show they are following along, without taking the floor.
- Agentic workflow — a task sequence where an AI system plans, uses tools and takes multiple actions over time, rather than giving one single response.
- Plug-and-play component — a module designed so it can be replaced with a different or upgraded version without retraining or redesigning the rest of the system.
- Graph-based orchestration — an architecture style where an application's data and model flows are represented as a graph of connected nodes that can be deployed and managed together.
- Streaming input — data such as audio or video that arrives continuously over time, rather than as a single complete file given all at once.
Sources
- Omni Interaction Agent Technical Report (Gander) — arXiv, 2026
- Gartner's Top Strategic Technology Trends for 2027: Speech Reasoning Models — Gartner, 2026
- How We're Building A Real-time Multi-modal AI Backend With Deployable Graphs — Gabber, 2026
- Gartner Predicts 80% of Enterprise Software and Applications Will Be Multimodal by 2030, Up from Less Than 10% in 2024 — Gartner, 2025
- Gartner Predicts 40% of Generative AI Solutions Will Be Multimodal By 2027 — Gartner, 2024