DM Explains · No. 023 · Orchestration
The multimodal orchestrator: why no one is training one giant model anymore
Instead of building one model that understands text, images, audio and video at once, teams are wiring together specialists through a controller that decides who does what. It's cheaper, and sometimes better.
· 15 min read
Read me first
Multimodal orchestration genuinely does beat training one giant omnimodal model, but only when the specialists are good enough on their own; the controller never watches a video or hears audio itself, it only decides who should.
Illustrative: a video with narration, captions and action, asked to summarise and flag contradictions → 1 detects speech, text, vision present → 2 splits into transcribe, OCR, caption-the-action → 3 routes each to a speech, text and vision expert → 4 runs the three in parallel since none depends on another → 5 all three write findings into one shared notebook → 6 controller stitches the notebook, flags that the narration says 'empty room' while the footage shows people, returns one grounded summary
A user drops a video with narration, on-screen text and background noise into a chat box and asks what's actually happening in it. One request, four kinds of content, one answer expected. Build a single model trained from scratch to understand all of that jointly, and the training bill is enormous, and the day the video-understanding part of it falls behind the state of the art, there is no clean way to swap just that piece out. Get the coordination wrong instead — hand sub-tasks to specialists but never check their outputs line up — and the system produces a confident, polished answer that quietly contradicts itself.
Everyday picture. Amma does not cook, clean and do the laundry herself for the evening meal. She tells Meera to chop vegetables, asks Papa to run the washing machine, and has Dadi fold clothes, then brings the finished pieces together for dinner.
Where the picture breaks. Amma trusts her family without a protocol; everyone already understands what "chopped" or "folded" means and what to do if something goes wrong. A real orchestrator has no such shared understanding — it has to formally check each specialist's output format and handle mismatches or outright failures, which a household rarely has to engineer for.
Why it exists now
Through 2026, several orchestration frameworks were published that generalise single-agent, single-modality pipelines into multi-agent, multi-modality ones. Orchestra-o1 reports a 10.3% accuracy gain over the next-best approach on the OmniGAIA benchmark, and a 32.8% improvement over the AOrchestra baseline when GPT-5 is used as the main agent (arXiv). Separately, a training-free approach to combining large multimodal models claimed performance gains of up to 7.8% and latency reductions of 10.3% against jointly-trained systems (OpenReview). These releases, alongside open projects such as MAC-AutoML's training-free orchestration code (GitHub), mark a shift away from narrow, single-modality agent frameworks and towards general-purpose orchestration as a cheaper, more flexible alternative to retraining a model from scratch every time a new modality or a better specialist shows up.
The thing that changed is not any one model getting smarter. It is that specialist models for vision, speech and video individually got good enough, and cheap enough to call, that stitching them together through an LLM controller started beating the economics and sometimes the accuracy of joint training.
How it works
Detect the mix
You can't route what you haven't named
What it is. A controller LLM looks at the incoming request and works out which modalities are actually present — text, image, audio, video, some combination — and what the user wants done with each.
Why it exists. Nothing downstream can be routed correctly until the system knows what kinds of content it is dealing with and what the goal is for each one.
Where it breaks. Misclassifying the mix early cascades: if the controller misses that there's background noise worth flagging, no later step recovers that signal.
How that's handled. Systems like the training-free MLLM orchestration approach have the controller emit explicit control tokens for expert selection and sequencing, so the detection step is auditable rather than a hidden guess (arXiv.science).
In technical terms
In technical terms: a controller LLM infers user intent and emits protocol-constrained control tokens that name both modality and required capability, rather than free-text instructions, before any expert is invoked.
Split the task
One big job becomes many small, named jobs
What it is. The controller breaks the overall request into sub-tasks, each tagged to a modality or capability: "transcribe audio", "caption visual frames", "extract on-screen text".
Why it exists. A single generalist model asked to do all of this at once has no natural seam at which to apply the right specialist; decomposition creates that seam explicitly.
Where it breaks. Over-splitting trivial tasks adds pointless latency; under-splitting complex ones loses nuance that a dedicated step would have caught.
How that's handled. Orchestra-o1's mechanism explicitly supports modality-aware task decomposition alongside online sub-agent specialisation and parallel sub-task execution (arXiv).
In technical terms
In technical terms: decomposition produces a set of sub-tasks each carrying a required capability tag, decoupling "what needs doing" from "which model will do it".
Route by capability
Swap the expert, keep the plumbing
What it is. Each sub-task goes to whichever model or tool declares the needed capability — a vision-language model, an automatic speech recognition model, a text-to-speech model — rather than to a hardcoded model name.
Why it exists. Routing by capability instead of by fixed model name lets a team upgrade or replace one expert without touching the rest of the pipeline.
Where it breaks. If two experts both claim the same capability but perform differently in practice, naive routing has no way to prefer one without extra signal.
How that's handled. Orchestral, a TypeScript orchestration layer, routes capability requests such as text-to-image or image-to-video to a caller-supplied model through an adapter of roughly 15 lines, with no provider software development kit or application programming interface keys shipped by the layer itself (GitHub). Vorch-Omni goes further, building more than 10 tasks on a single flow-matching diffusion transformer without task-specific architectural changes (arXiv).
In technical terms
In technical terms: a capability registry maps task types to candidate models, and a thin adapter translates the orchestrator's generic call into whatever interface the specific model actually exposes.
Run in parallel or chain
Independent work runs together, dependent work waits
Illustrative: video with narration, captions and action, asked to flag contradictions → 1 split into speech, vision, text sub-tasks → 2 none depend on another → 3 vision and speech run in parallel → 4 (skipped, no dependency) → 5 outputs merged into a shared text notebook → 6 narration claim checked against visual action, contradiction flagged → coherent, traceable answer returned
What it is. A planner decides whether sub-tasks run as a parallel block, a sequential chain, or something interleaved, based on whether one step's output is needed before the next can start.
Why it exists. Transcribing audio and captioning video frames don't depend on each other, so running them side by side saves time; checking whether narration contradicts what's shown on screen needs both finished first.
Where it breaks. A plan fixed at the start can't adapt mid-task if an early result changes what's actually needed next.
How that's handled. The Chain of Modality framework uses a Planner that adaptively constructs a task-specific modality chain, choosing between Parallel, Sequential and Interleaved execution units rather than committing to one topology for every task (arXiv). The same paper describes conventional fixed-pipeline orchestration as relying on "passive joint fusion" with "static inertia" — a direct criticism of systems that don't adapt their topology to the task.
In technical terms
In technical terms: the planner selects an execution topology — parallel, sequential or interleaved — per task, rather than hardcoding one shape for every request.
Share memory
Keep everything in one shared, readable notebook
What it is. Instead of re-passing raw audio or video between steps, a text-centric memory stores transcripts, captions and structured facts that any later step can retrieve cheaply.
Why it exists. Raw multimodal data is heavy to move around and re-process; compressed text records are not.
Where it breaks. Compressing to text loses some of the raw signal — a subtle tone of voice or a visual detail that doesn't translate cleanly into words can be flattened out of existence.
How that's handled. The training-free MLLM orchestration approach uses exactly this kind of text-centric cross-modal memory, compressing multimodal evidence into lightweight structured records for retrieval and reuse across turns (arXiv.science).
In technical terms
In technical terms: cross-modal memory stores structured text records keyed to the originating sub-task, letting later steps retrieve prior evidence without re-invoking the specialist that produced it.
Synthesise and align
Stitching pieces into one answer, and training to do it well
What it is. A synthesis step merges the sub-agents' outputs into one final response, reconciling anything that conflicts between them.
Why it exists. None of the specialists see the whole picture; something has to reconcile a transcript that says one thing with a caption that shows another.
Where it breaks. Synthesis logic still has to resolve genuine contradictions between modalities, and that is exactly where errors surface most visibly to a user.
How that's handled. Orchestra-o1-8B is trained with decision-aligned group relative policy optimisation, an agentic reinforcement learning approach, to align the orchestrator's own routing and decomposition decisions with final task success, reporting state-of-the-art performance among open-source omnimodal agents on this measure (arXiv).
In technical terms
In technical terms: decision-aligned group relative policy optimisation (DA-GRPO) trains the orchestrator's policy by rewarding the sequence of routing decisions that led to a correct final answer, not just the final answer in isolation.
Compared with what came before
| One jointly-trained omnimodal model | Multimodal orchestrator | |
|---|---|---|
| How it's built | Trained from scratch on paired data across all modalities | Assembled from existing specialist models, usually without further training |
| Upgrading one modality | Requires retraining or fine-tuning the whole model | Swap the specialist behind the capability registry |
| Transparency of decisions | Routing logic hidden inside one set of weights | Decomposition and routing are explicit, inspectable steps |
| Latency on independent sub-tasks | Fixed, whatever the model's internal path is | Can run genuinely in parallel when sub-tasks don't depend on each other |
| Cost to build | Large pretraining cost up front | Cost shifts to orchestration logic and per-call specialist fees |
| Failure mode | Opaque — wrong answers with no visible cause | Visible — a traceable sub-task or synthesis step to blame |
The orchestrator's chief advantage is not that it is smarter than a jointly-trained model; several of the dossier's results, including M*'s reported "on average 20% lower end-to-end latency than vLLM-Omni for text-to-image workloads on BAGEL" and "up to 2.9x lower real-time factor plus 2.7x higher throughput for text-to-speech workloads on Qwen3-Omni" (DeepPaper mirror of arXiv), are about serving efficiency rather than raw capability. The advantage is that nobody has to retrain anything to get there, and when something goes wrong, there's a step to point at.
Where it breaks
Orchestration genuinely helps when the task decomposes cleanly into modality-specific pieces that specialists already handle well; it struggles when the modalities interact in ways no decomposition step anticipated, or when every extra hop adds latency a user will notice.
Not a substitute for a weak specialist
Routing by capability only works if the specialist being routed to is actually good at that capability. If the available speech recognition model is poor, no amount of clever orchestration logic repairs its transcription errors before they propagate into the final synthesis step.
Fixed-topology pipelines, the kind the Chain of Modality paper criticises for "passive joint fusion" and "static inertia" (arXiv), fail quietly: they keep running, they just don't adapt the execution shape to the task, so a request that needed an interleaved chain gets forced through a rigid sequential one and loses accuracy without any obvious error message. Latency compounds across hops too — every sub-task invocation is a network call to a separate model, and that adds up unless the planner is disciplined about running independent work in parallel rather than one step at a time.
Numbers to carry
| Measure | Value |
|---|---|
| Orchestra-o1 vs next-best approach, OmniGAIA benchmark | 10.3% accuracy gain |
| Orchestra-o1 vs AOrchestra baseline (GPT-5 as main agent) | 32.8% improvement |
| Orchestra-o1 vs Gemini-3-Pro (GPT-5 as main agent) | 10.3% improvement |
| Training-free MLLM orchestration vs jointly-trained systems | up to 7.8% performance gain, 10.3% lower latency |
| M* vs vLLM-Omni, text-to-image on BAGEL | average 20% lower end-to-end latency |
| M* vs vLLM-Omni, text-to-speech on Qwen3-Omni | up to 2.9x lower real-time factor, 2.7x higher throughput |
| One Supervisor, Many Modalities (2,847 queries, 15 task categories) | 72% reduction in time-to-accurate-answer, 85% reduction in conversational rework |
| Vorch-Omni, single flow-matching diffusion transformer | supports more than 10 tasks without architectural changes |
| Orchestral adapter length | roughly 15 lines, no provider software development kit shipped |
Myths worth killing
| Commonly repeated | What is actually true |
|---|---|
| "Orchestration is just a model router" | It decomposes tasks across modalities and manages parallel or sequential sub-agent execution with shared memory, not just forwarding one request to one model |
| "You still need to train something" | Several of these systems, including the training-free MLLM orchestration approach, work with no additional model training at all |
| "One big model will always understand context better" | On the OmniGAIA benchmark, Orchestra-o1 outperformed both an agent-swarm baseline and a leading proprietary model, Gemini-3-Pro, by 10.3% |
| "Parallel execution is always faster" | The planner chooses parallel only when sub-tasks are independent; dependent steps, like checking narration against visuals, must wait for both inputs |
| "This is just agent swarms with extra steps" | Agent swarm frameworks were largely limited to text-only agents; multimodal orchestrators extend the same coordination idea to image, audio and video specialists |
Worked example
- User submits one video file with spoken narration, on-screen captions and visual action, asking the system to summarise what's happening and flag any contradiction between what's said and what's shown.
- Controller detects three modalities in play — audio, visual frames, on-screen text — and infers two intents: summarise, and cross-check for contradiction.
- Controller splits the request into sub-tasks: transcribe audio, caption visual frames, extract on-screen text, then a fourth sub-task to compare the three outputs.
- Router sends transcription to a speech recognition specialist, frame captioning to a vision-language model, and text extraction to a text-recognition tool, each matched by capability tag rather than by name.
- Planner runs the first three sub-tasks in parallel, since none depends on another; it holds the comparison sub-task until all three finish.
- Memory stores each specialist's output as a short text record — a transcript, a set of captions, extracted on-screen strings — rather than keeping the raw audio or video around for later steps.
- Synthesiser reads the three text records, finds (illustrative) that the narration claims "the device shuts off automatically" while the captioned frames show a hand manually flipping a switch, and flags the contradiction in the final answer alongside the summary.
Should you use it?
Reach for a multimodal orchestrator when a task genuinely mixes modalities that need to be understood together, when the team wants to reuse best-in-class specialist models rather than retrain something new, or when the routing decisions themselves need to be auditable rather than buried inside one opaque model. Skip it for simple single-modality tasks where one text-only or vision-only model already does the job, for situations where the multi-step routing overhead is unacceptable for latency reasons, or where deployment simplicity on constrained hardware favours shipping one set of weights over coordinating several.
The cheapest first experiment is not building a full orchestration stack. It's taking one existing multimodal task already running on a single generalist model, splitting it manually into two sub-tasks routed to two separate specialists, and measuring whether the combined result beats the generalist on the parts that mattered — before investing in a planner, a capability registry or shared memory.
Say it in the room
"Use it when the task mixes modalities a single model handles unevenly; skip it when one specialist model already does the whole job well enough."
Lines to carry
- "Don't train one mind to do everything; hire specialists and coordinate them."
- "Route by what a model can do, not by its name."
- "Parallel when nothing depends on anything else, sequential when it does."
- "A shared notebook in plain text beats passing raw files around."
- "Orchestration makes decisions visible; one giant model hides them."
- "The controller never touches raw pixels or waveforms itself — only decides who should."
FAQ
Isn't this just calling several APIs and gluing the results together?
That's close to the mechanism, but the gluing is the hard part. The controller has to detect which modalities are even present, decide how to split the task, pick the right specialist by capability rather than by a fixed name, decide whether sub-tasks can run at the same time or must wait on each other, and then reconcile outputs that might contradict each other. Each of those is a step that can go wrong on its own.
Does an orchestrator need any training at all?
Not necessarily. The training-free MLLM orchestration approach reports performance gains of up to 7.8% over jointly-trained systems with no additional training (OpenReview). Other systems, like Orchestra-o1-8B, do add reinforcement learning specifically to improve the orchestrator's own routing decisions, which is a different thing from training a joint multimodal model from scratch.
What happens if two specialists disagree?
That's handled in the synthesis step, where the orchestrator reconciles sub-agent outputs into one final answer. It's also the step most likely to visibly fail, because genuine contradictions between modalities — narration saying one thing, video showing another — have to be resolved by logic that sits above both specialists, not inside either of them.
Is this faster than one big model, or just more flexible?
Both, in some reported cases. M* reports on average 20% lower end-to-end latency than vLLM-Omni for text-to-image workloads on BAGEL, and up to 2.9x lower real-time factor for text-to-speech on Qwen3-Omni (DeepPaper mirror of arXiv). But the main draw is flexibility — swapping one specialist without retraining everything — and speed gains depend heavily on the workload.
Can I swap one specialist model for a better one later?
That's the point of capability routing. Orchestral's adapters run to roughly 15 lines and ship with no provider software development kit, which is designed specifically so a caller can plug in whichever model handles a given capability best, and change it later without rewriting the orchestration logic itself (GitHub).
Glossary
- Orchestrator — a controller, usually an LLM, that decides how a task should be broken up and which specialist models or tools should handle each part.
- Multimodal / omnimodal — involving more than one kind of input or output, such as text, images, audio and video, handled together.
- Agent swarm — a system made of multiple AI agents working together, each potentially specialised for a sub-task, coordinated to complete a larger goal.
- Sub-agent — a smaller, specialised AI component invoked by the orchestrator to handle one specific part of a task.
- Capability routing — directing a task to whichever model or tool can perform a required function, rather than hardcoding which model to call.
- Training-free integration — combining existing pretrained models into a working system without further gradient-based training to align them.
- Cross-modal memory — a shared store of information, often converted into text, that lets the system recall facts or context gathered from different modalities.
- Benchmark (e.g. OmniGAIA) — a standardised set of test tasks used to measure and compare how well different AI systems perform.
- Reinforcement learning (RL) — a training method where a system learns by receiving rewards or penalties based on the outcomes of its decisions.
- DA-GRPO — decision-aligned group relative policy optimisation, a reinforcement learning technique used to train Orchestra-o1-8B to make better orchestration decisions.
- Flow-matching diffusion transformer — a generative neural network architecture used to produce outputs like video or audio by gradually refining random noise into a target output.
- Latency — the time delay between a request being made and the system producing its response.
- Real-time factor — a measure of how a system's processing speed compares to the actual duration of the audio or video it is generating or processing.
- ASR / TTS — Automatic Speech Recognition (converting speech to text) and Text-to-Speech (converting text to spoken audio).
- Vision-language model (VLM) — a model that can process both images and text together, such as to answer questions about an image.
Sources
- Orchestra-o1: Omnimodal Agent Orchestration — arXiv, 2026
- Training-Free Multimodal Large Language Model Orchestration — arXiv.science
- Vorch-Omni: Multi-Task Orchestration of Sight and Sound — arXiv, 2026
- MAC-AutoML/Trainingfree-LLM-Orchestration — GitHub
- Chain of Modality: Adaptive Modality Orchestration Framework — arXiv, 2026
- Training-Free Multimodal Large Language Model Orchestration — OpenReview
- orchestral-media/orchestral: TypeScript orchestration layer for media generation — GitHub
- One Supervisor, Many Modalities: Adaptive Tool Orchestration for Autonomous Queries — arXiv, 2026
- M*: A Modular, Extensible, Serving System for Multimodal Models — DeepPaper (arXiv mirror), 2026