DM Explains · No. 014 · Orchestration
Multimodal orchestration layers: why AI stopped trying to be one brain
Instead of training one giant model to see, hear and speak, the newest systems route each task to a specialist and stitch the results together with a language model playing traffic controller.
· 13 min read
Read me first
Multimodal orchestration genuinely does let a system see, hear and speak without training one giant model to do all three, but only when the task is long enough or complex enough to pay for the handoffs between specialists.
The fridge-photo request: 1 controller sees voice+photo, routes to vision then text → 2 vision expert reads photo, text expert reasons about what's missing → 3 result compressed to a text record, not the raw image → 4 speech layer stays live for follow-up without restart → 5 the graph schedules vision before text before speech → 6 the KV cache from the vision call is reused when speech reads the list back
Amma asks the family voice assistant to look at a photo of the fridge, read out what's missing for dinner, and add it to the shopping list — three different jobs in one breath. Build this as one monolithic model and every upgrade to the vision half means retraining the whole thing; get the handoffs between specialists wrong and the assistant forgets what it just saw by the time it starts talking.
Everyday picture. Amma does not cook every dish herself: she tells Dadi to make the rice, Papa to fry the vegetables, Arjun to set the table, then brings it all together on the table at the right moment.
Where the picture breaks. Amma could, in principle, learn to cook every dish herself from scratch — like a jointly-trained model fusing all modalities in one set of weights — but orchestration deliberately skips that, trading some depth of integration for the freedom to swap out any one cook.
Why it exists now
Training one model that natively reads images, hears speech, writes text and generates pictures requires enormous joint training runs, and every time a better vision model or speech model appears, the whole fused system has to be retrained to use it. Through 2026 several research groups published a different answer: keep the specialist models separate, and put a language model in charge of routing between them. The ICML 2026 poster on Training-Free Multimodal Large Language Model Orchestration, a GitHub implementation under the MAC-AutoML project, the Omni-Flow serving framework described on arXiv, and a separate supervisor-based orchestration paper evaluated on 2,847 queries all appeared within months of each other, treating orchestration as a distinct engineering layer with its own benchmarks and tooling rather than a one-off wrapper script written to glue two APIs together.
What changed is less a single breakthrough than a convergence: language models became reliable enough to act as controllers that emit structured instructions rather than free text, specialist models for vision, speech and image generation matured independently, and serving infrastructure caught up with mechanisms — shared caches, dependency graphs — that make routing between heterogeneous models cheap enough to be worth doing.
How it works
Intent routing
One controller decides who does what, and in what order
What it is. An LLM reads the incoming request and decides, before anything else happens, which specialist models need to be involved and in what sequence.
Why it exists. Without a central decision-maker, nothing in the system knows whether a request needs a vision model, a speech model, or just plain text generation — this step determines the entire execution path.
Where it breaks. If the controller misreads intent, the wrong expert gets called and the mistake propagates downstream before anyone notices.
How that's handled. The controller's choice is made explicit and auditable rather than hidden inside one network, so a wrong routing decision can be inspected and corrected rather than being buried in weights.
In technical terms
In technical terms: the Training-Free Multimodal LLM Orchestration framework uses an LLM controller that infers user intent and emits explicit control tokens selecting a modality expert and a sequence, keeping the routing decision protocol-constrained and auditable.
Expert invocation
Specialists do one job well instead of everything badly
What it is. Whichever specialist got picked — object detection, optical character recognition, speech transcription, a diffusion image generator, text-to-speech — gets called through its own existing interface.
Why it exists. Reusing already-trained experts avoids retraining a single model to handle every modality, and lets each expert be swapped out independently as better versions appear.
Where it breaks. Experts trained separately never develop the deep cross-modal correlations a jointly-trained model could learn between, say, exactly what a word sounds like and exactly what it looks like written down.
How that's handled. The MAC-AutoML implementation lets each expert be replaced or upgraded through configuration rather than retraining, which is the whole point: swappable beats seamless, until seamless is what the task actually needs.
In technical terms
In technical terms: the framework integrates off-the-shelf modality experts into a unified multimodal input-output system without any additional gradient-based training for integration, according to the ICML poster.
Memory compression
Remember the gist in text, not the raw picture
What it is. What each expert found gets written down as a compact text record rather than kept as a raw image, audio clip or tensor.
Why it exists. This lets later turns in a conversation retrieve prior evidence by a cheap text lookup instead of re-running a vision or audio model on the same input again.
Where it breaks. Compression loses detail that the raw audio or video retained, and someone has to decide in advance what's worth keeping.
How that's handled. The text-centric cross-modal memory in the orchestration framework is designed specifically to reduce redundant expert invocations across turns, trading some fidelity for reuse.
In technical terms
In technical terms: a text-centric cross-modal memory compresses multimodal evidence into structured records for lightweight retrieval, according to the same ICML 2026 poster.
Live interaction layer
Switching modalities mid-sentence without restarting
What it is. A runtime that handles streaming audio, interruptions, and handoffs between modalities while a conversation is still happening, rather than only between discrete requests.
Why it exists. A real assistant needs to react mid-response — react to an interruption, switch from speech to grounding against live video — without restarting the whole pipeline each time.
Where it breaks. This machinery is overkill for a single-shot text query; the orchestration overhead itself can cost more than it saves when nothing actually needs to be streamed.
How that's handled. It is scoped to live, multi-turn, multimodal sessions, where full-duplex streaming and interruption-aware dialogue are the whole point rather than an afterthought.
In technical terms
In technical terms: the unified interaction layer supports full-duplex streaming, speech interruption, live video grounding and consistent modality transitions, per the MAC-AutoML documentation.
Workflow scheduling
A graph decides what can run when
What it is. At serving scale, the whole request is laid out as a graph of components — encoders, language models, diffusion models, decoders — rather than a fixed pipeline.
Why it exists. Production serving needs to schedule heterogeneous components with different hardware needs and latencies independently, instead of forcing everything through one rigid sequence.
Where it breaks. Graph scheduling adds coordination cost that only pays off at production scale with many concurrent, heterogeneous jobs running at once.
How that's handled. The runtime executes each component once its dependencies in the graph are satisfied, which is what makes concurrent heterogeneous jobs possible in the first place.
In technical terms
In technical terms: Omni-Flow's Control Flow layer defines workflows through a Python domain-specific language, organising heterogeneous components into a unified dataflow graph that the runtime executes based on dependency satisfaction.
Cache sharing
Don't recompute what was already computed
What it is. Intermediate computations — key-value caches — get reused across conversation turns and even across different model types on the same hardware.
Why it exists. Sharing cached computation avoids resubmitting an entire conversation's accumulated context on every single turn, cutting both GPU memory use and latency.
Where it breaks. In short, single-turn sessions there is nothing accumulated to reuse, so the caching machinery adds coordination latency instead of cutting it.
How that's handled. Omni-Flow's Data Flow layer manages placement, transfer and lifetime of shared tensors and paged caches across roles, including reuse of prefix computations between a language model and a diffusion model sharing a device.
In technical terms
In technical terms: Omni-Flow enables cross-role prefix key-value cache reuse between compatible LLM and diffusion components, alongside same-device weight sharing, as described in the arXiv paper.
Compared with what came before
| One jointly-trained omni model | Multimodal orchestration layer | |
|---|---|---|
| Adding a better vision model | Requires retraining the whole fused model | Swap the expert through configuration |
| Cross-modal correlation depth | Learned jointly inside one set of weights | Limited to what routing and shared text memory can capture |
| Cost to extend | High: joint training runs | Lower: no additional joint training needed for integration |
| Best suited to | Deep nuanced reasoning across modalities at once | Composable, swappable pipelines with independent upgrade cycles |
| Serving concern | One model, one serving stack | Heterogeneous components needing a dependency graph and shared caches |
The judgement underneath the table is simple: orchestration buys composability and independent upgrades at the price of the deep, jointly-learned correlations a fused model can capture. Which trade-off wins depends entirely on whether the task needs that depth or just needs the pieces stitched together competently.
Where it breaks
Orchestration genuinely cuts cost and development time when the task is composable, but the gains are conditional on session length and task complexity, not automatic.
Overhead can exceed the gains outside long sessions
Omni-Flow's own aggregate job completion time across four benchmarks and multiple concurrency levels was reported as 1.2% higher than vLLM-Omni's and 4.9% higher than SGLang-Omni's in standard, non-multi-turn settings — the coordination machinery costs more than it saves when there is nothing accumulated to reuse. The gains the same paper reports — a 7.5% and 19.3% reduction in mean job completion time against vLLM-Omni and SGLang-Omni respectively — only showed up in a 50-turn multimodal session, where avoiding resubmission of accumulated inputs actually pays off.
Beyond latency, routing itself is a single point of failure: a misread intent sends the whole request down the wrong path before any expert has a chance to correct it, and because the system does not understand the request, it has no way of noticing its own mistake. And text-centric memory, by design, throws away whatever detail does not make it into the compressed record — fine for a shopping list, not fine for a task that needs the raw waveform back later.
Numbers to carry
| Figure | Value |
|---|---|
| MAC-AutoML performance improvement over jointly-trained approaches | up to 7.8%, as reported in the framework's paper (2026) |
| MAC-AutoML latency reduction | 10.3%, same source |
| Omni-Flow job completion time vs vLLM-Omni / SGLang-Omni (standard, non-multi-turn) | 1.2% higher / 4.9% higher (2026) |
| Omni-Flow job completion time vs vLLM-Omni / SGLang-Omni (50-turn session) | 7.5% lower / 19.3% lower (2026) |
| Supervisor-based framework: queries evaluated | 2,847 queries across 15 task categories |
| Supervisor-based framework: reductions vs hierarchical baseline | 72% reduction in time-to-accurate-answer, 85% reduction in conversational rework, 67% cost reduction (2026) |
Myths worth killing
| Commonly repeated | What is actually true |
|---|---|
| "Orchestration always beats one big model" | Omni-Flow's own measurements show it can be slightly slower than plain serving stacks outside long multi-turn sessions. |
| "Experts need to be retrained to work together" | The ICML framework and the MAC-AutoML implementation integrate off-the-shelf experts with no additional joint training for integration. |
| "The controller understands the request" | The controller is a language model emitting control tokens based on patterns in the request; it predicts a routing decision, it does not comprehend intent the way a person would. |
| "Memory compression keeps everything" | The cross-modal memory stores compressed text records specifically to reduce redundant expert calls, which means detail from the original image, audio or video is deliberately not retained. |
Worked example
- Amma speaks the request: "What's missing from the fridge for dinner, and add it to the list."
- The controller (an LLM) reads the request and emits control tokens routing the image to a vision expert and the final step to a text-to-speech expert, following the pattern described in the ICML 2026 poster.
- The vision expert reads the fridge photo and reports back, say, "no onions, no yoghurt" — illustrative, since the exact contents aren't in any dossier fact.
- The text controller reasons over that output and the family's usual dinner plan to decide what belongs on the shopping list.
- Cross-modal memory stores "fridge check, 2026: missing onions, yoghurt" as a compact text record rather than keeping the photo itself, so a later question like "did we already check the fridge today" can be answered without calling the vision expert again.
- The text-to-speech expert is invoked last, reading the finished list back to Amma.
- If Amma interrupts mid-reply to ask something else, the live interaction layer is what lets the system react without restarting steps 2 through 6 from scratch.
Should you use it?
Orchestration earns its keep when an application genuinely needs several distinct modalities working together without retraining a unified model, when the team wants to swap specialist models independently as better ones appear, or when sessions are long and multi-turn enough that cached context actually gets reused. It is not for tasks needing the kind of nuanced cross-modal reasoning that only joint training captures, nor for latency-sensitive single-modality tasks where the routing overhead itself is the problem, nor for small deployments where one existing multimodal model already does the job without needing to be composable.
The cheapest first experiment is not building the full scheduling and cache-sharing stack: it is wiring an LLM controller to route between two existing experts for a narrow task, and measuring whether the routing decisions are reliable before any of the serving-layer machinery is worth adding.
Say it in the room
"Use orchestration when the task needs several specialists stitched together and must stay upgradeable; skip it for single-shot, single-modality work where the routing overhead costs more than it saves."
Lines to carry
- "One controller, many specialists, no retraining."
- "Memory stored as text travels further than memory stored as pixels."
- "Orchestration overhead is a tax paid only once the session gets long."
- "Swappable beats seamless, until seamless is what the task actually needs."
- "The controller routes the request; it does not understand it."
FAQ
Is this the same as an LLM agent calling tools?
Not quite. Generic agent frameworks call arbitrary external tools for mostly single-turn tasks. Multimodal orchestration is narrower and more specific: it manages modality experts, compresses cross-modal memory, and handles real-time streaming and interruptions across simultaneous input and output channels, which a generic tool-calling setup typically does not attempt.
Does orchestration need any training at all?
The routing and integration itself does not — the ICML 2026 framework and the MAC-AutoML implementation explicitly avoid additional gradient-based training to wire experts together. The individual experts, of course, were trained separately beforehand; "training-free" refers to the glue, not the ingredients.
Why would a system be slower with orchestration than without it?
Because routing, memory lookups and cache coordination all cost something. Omni-Flow's own figures show it running 1.2% and 4.9% slower than two existing serving stacks in standard, non-multi-turn settings — the coordination overhead only pays for itself once there's enough accumulated context across turns to reuse.
What happens if the controller routes a request to the wrong expert?
The mistake propagates: a vision task sent to the wrong pipeline won't be caught downstream, because nothing in the system is checking the controller's reasoning against ground truth. This is why the routing decision is kept explicit, as control tokens, rather than buried inside a single network — at least a wrong decision can be inspected afterwards.
Does the shared memory remember everything the system has seen?
No — it is deliberately compressed into text records, which means detail present in the original image, audio or video is not kept. The trade-off is reuse without recomputation, not a complete archive.
Glossary
- Multimodal — involving more than one type of data or input/output, such as text, images, audio and video together.
- Orchestration layer — a coordination system that decides which specialised model handles which part of a task and combines their outputs.
- LLM controller — a large language model used as the decision-making component that interprets requests and issues instructions to other models.
- Training-free — describes a system built by combining existing pre-trained models without further gradient-based training to integrate them.
- Control tokens — special symbols emitted by a controller model that instruct the system which expert to call or what action to take next.
- Cross-modal memory — a shared storage system that keeps compressed records of information gathered from different modalities so it can be reused later.
- Full-duplex streaming — the ability of a system to send and receive data at the same time, rather than one after another.
- KV cache (key-value cache) — stored intermediate computations from a language model that can be reused to avoid recomputing the same context repeatedly.
- DAG (dataflow graph) — a directed acyclic graph representing tasks and their dependencies, used to decide execution order.
- DSL (domain-specific language) — a small programming language designed for a specific purpose, here for defining multimodal workflows.
- Job completion time (JCT) — the total time taken for a computing task to finish, used as a performance metric for serving systems.
- Supervisor (in agentic frameworks) — a central component that breaks down a user's request into smaller subtasks and assigns them to specialised tools.
Sources
- Training-Free Multimodal Large Language Model Orchestration — ICML 2026, 2026
- Trainingfree-LLM-Orchestration README — GitHub (MAC-AutoML)
- Omni-Flow: A Unified Workflow Orchestration and Distributed KV Cache Sharing Framework for Multimodal Inference — arXiv, 2026
- An Agentic AI Framework for Autonomous Multimodal Query Processing — arXiv, 2026
- Training-Free Multimodal Large Language Model Orchestration (Paper page) — Hugging Face, 2025