DM Explains · No. 031 · Orchestration
Multi-modal orchestration: why one supervisor beats one giant model
Instead of training one model that sees, hears and reads, you can wire together specialists that already do each job well — and a controller that decides who answers what.
· 13 min read
Here is the corrected article:
Read me first
Multi-modal orchestration genuinely does let a system see, hear and reason without one enormous model trained to do all three — but only when a controller routes well, and routing mistakes cost a round trip, not an answer.
The assistant receives a photo of a broken appliance and a spoken question about warranty coverage → 1 controller routes image and audio → 2 vision model reads the photo, transcriber reads the speech → 3 both call tools through the same MCP server → 4 each runs in its own sandboxed phase, not one giant prompt → 5 results and prior interactions are written to a shared notebook → 6 a human approves the draft before 'Yes, covered under warranty' is sent
A family wants to ask their home assistant a question while holding up a photo of a broken appliance and talking over the dog barking. Text, image and speech arrive at once, and something has to decide what happens to each piece before anyone gets an answer.
Build that something as one giant model trained jointly on images, audio and text, and you are waiting on expensive joint training before the system can improve at all. Build it as a pile of bolt-on API calls with no coordination, and context gets dropped between turns, tools get called by the wrong agent, and nobody can say afterwards which part of the system decided what.
Everyday picture. Amma stands in the kitchen doorway as supervisor, calling out jobs rather than cooking everything herself: Papa fries the snacks, Dadi plates the sweets, Arjun answers the door — each handling one kind of task while Amma coordinates and decides what happens next.
Where the picture breaks. Amma can improvise new instructions on the spot in ways a software controller cannot; the controller must follow pre-defined routing logic and protocols, not judgement calls.
Why it exists now
Training one model that handles vision, speech and text jointly is slow and expensive, and it locks you into whatever specialists were bundled in at training time. The alternative that has matured is to leave the specialists as they are and coordinate them at the moment a request arrives.
Open-source projects such as ForgeFlow have published architectures that route work between a supervisor agent and specialist agents — researcher, analyser, executor — through the Model Context Protocol, gated by human approval before anything costly runs. Academic work has gone further: the Training-Free Multimodal Large Language Model Orchestration framework describes a controller LLM that analyses incoming inputs and dynamically routes tasks to specialised models through designed agents, with no additional joint training at all. What changed is not a single model getting bigger; it's a protocol and a controller pattern becoming reliable enough to wire specialists together without retraining anything.
How it works
Controller routes
One voice decides who answers what
What it is. A supervisor model reads the whole incoming request and assigns each piece to a specialist rather than attempting everything itself.
Why it exists. Without a dispatcher, there is no way to send an image to a vision model and a spoken question to a transcriber in the same request.
Where it breaks. If the controller misjudges which modality dominates the request, it can send the wrong piece to the wrong specialist, costing a wasted round trip rather than a wrong answer outright.
How that's handled. The controller's routing decision is logged, so a misroute shows up as a visible step rather than a silent failure.
In technical terms
In technical terms: the MLLM Orchestration framework uses a central controller LLM that analyses user inputs and dynamically routes tasks to specialised models through carefully designed agents.
Perception split
Each sense gets its own specialist
What it is. Images, speech and text are each handled by a model purpose-built for that input, run independently before the results are merged.
Why it exists. A model purpose-built for reading images or transcribing speech tends to do that one job better than a single model asked to do everything at once.
Where it breaks. Deep fusion of senses learned jointly during training — the kind where seeing and hearing shape each other from the start — can't be replicated by specialists that never saw each other's inputs during training.
How that's handled. The controller merges outputs after the fact rather than pretending the specialists shared context they never had.
In technical terms
In technical terms: ForgeFlow supports multi-modal input through PDF text extraction and vision-LLM image description, implemented as a dedicated multimodal module, and a related tutorial describes pairing a vision model such as DAM-3B to analyse image regions within a multi-agent workflow.
Shared protocol
One desk, every tool checked out the same way
What it is. Agents reach tools and data only through a standard protocol, not bespoke wiring built per agent.
Why it exists. Standardising tool access means a new specialist agent can be added without rewriting the orchestration layer each time.
Where it breaks. A protocol enforces one transaction pattern at a time; it doesn't by itself guarantee that agents calling tools in parallel won't collide on shared state.
How that's handled. ForgeFlow's agents reach the outside world only through an MCP tool server, so every tool call is visible and auditable in one place.
In technical terms
In technical terms: a tutorial on multimodal human-AI interaction systems uses the Model Context Protocol to define how agents access shared tools and maintain consistent context across modalities.
Graph execution
Checkpoints, not one giant prompt
What it is. The workflow runs as a checkpointed graph, with each step its own sandboxed agent run rather than one sprawling prompt.
Why it exists. Phase-by-phase execution with checkpoints allows recovery and auditing of a failed step without restarting the whole task.
Where it breaks. Checkpointing adds overhead; for latency-critical real-time interaction, the extra round trips between nodes can cost more than they save.
How that's handled. ForgeFlow's execution engine orchestrates phase-by-phase, giving each top-level node its own agent run in its own sandbox with only the skills and inputs it needs, instead of compiling the whole flow into one prompt.
In technical terms
In technical terms: LangGraph is described as a scalable graph-based framework for orchestrating multi-agent systems with stateful, turn-based structures.
Shared memory
A notebook everyone can flip back through
What it is. A memory store keeps context across modalities and turns so agents aren't reasoning from a blank slate each time.
Why it exists. Coherent multi-turn, multi-modal conversation needs retrieval of earlier visual, spoken or textual context, not treating each turn as isolated.
Where it breaks. Retrieval is explicit and structured; it can miss the kind of nuance a person carries without even trying to remember it.
How that's handled. Entries are scoped so retrieval returns relevant context by meaning rather than exact wording.
In technical terms
In technical terms: ForgeFlow implements semantic memory using pgvector cosine recall scoped by namespace, while the MLLM Orchestration framework stores structured interaction histories in a standardised JSON format to maintain coherent context across modalities.
Approve?
human checks first
What it is. High-impact actions proposed by an agent wait for a person before anything executes.
Why it exists. Orchestrated agents can take real actions through tools, so a gate reduces the risk of unsupervised, costly or harmful actions.
Where it breaks. Gating adds latency and friction; overused, it turns an autonomous system back into a manual one that merely asks permission constantly.
How that's handled. ForgeFlow routes a supervisor agent's work to specialist agents and gates high-impact actions behind explicit human approval before execution.
In technical terms
In technical terms: the approval gate sits between the controller's proposed action and the tool server, so nothing costly runs without a recorded human decision.
Compared with what came before
Illustrative: a user shows a cracked blender and asks aloud "is this covered?" → 1 supervisor splits image and audio → 2 vision specialist reports a cracked base → 3 speech specialist transcribes the warranty question → 4 both descriptions written to a shared notebook via pgvector recall → 5 reconciliation checks the two agree → 6 document check finds the warranty clause → reply: "Yes, covered for 90 more days", 7.8% more accurate and 10.3% faster than a jointly-trained omni-model
| Jointly-trained omni-model | Training-free orchestration | |
|---|---|---|
| How it's built | One model trained from the outset on all modalities together | Existing specialist models wired together by a controller at request time |
| Adding a new skill | Requires retraining or fine-tuning the whole model | Add a new specialist agent behind the same protocol |
| Cross-modal fusion | Learned jointly during training, potentially deeper | Reconstructed after the fact by merging separate outputs |
| Reported performance | Baseline in the MLLM Orchestration paper's comparison | Up to 7.8% improvement over jointly-trained approaches and 10.3% lower latency, reported in the Training-Free Multimodal Large Language Model Orchestration paper |
| Example system | Gander, described as unifying omni perception, realtime interaction and agentic capability in a single framework via a Cerebellum-Brain collaborative architecture, per the Omni Interaction Agent Technical Report | ForgeFlow's hub-and-spoke system, with agents reaching tools only through an MCP server |
The orchestrated approach gives up the kind of fusion that only joint training can produce, but it gains modularity: specialists can be swapped or added without retraining anything, and in the one benchmark comparison in this dossier, it beat the jointly-trained baseline rather than merely matching it.
Where it breaks
Orchestration tends to hold up when several existing specialist models already perform well individually and need combining without retraining — it tends to fail when the task genuinely needs deep cross-modal fusion that only joint training produces, or when latency budgets are too tight for multiple round trips between agents.
Multi-tenant isolation is not finished in at least one real system
ForgeFlow's own documentation notes that multi-tenant query scoping via workspace_id is still "in progress", meaning tenant isolation in the orchestration layer is not yet complete in that deployed system. A protocol standardising tool access does not by itself guarantee that one tenant's data stays walled off from another's.
Real-time, full-duplex interaction — listening and speaking at once, or handling interruption — is harder to deliver through several separate models calling each other than through a single model built for that mode from the start; the extra round trips between agents introduce delay that careful engineering must claw back.
Numbers to carry
| Figure | Value |
|---|---|
| Performance gain over jointly-trained approaches reported by the MLLM Orchestration framework | up to 7.8%, as of 2026 |
| Latency reduction reported by the same framework | 10.3%, as of 2026 |
| ForgeFlow memory recall mechanism | pgvector cosine recall, scoped by namespace |
| ForgeFlow tenant isolation status | query scoping via workspace_id noted as "in progress" |
| ForgeFlow deployment targets | Docker Compose, Kubernetes, Helm, Terraform (AWS), Fly deployment target, and an air-gapped offline bundle |
| Example vision model named in a multimodal tutorial | DAM-3B |
Myths worth killing
| Commonly repeated | What is actually true |
|---|---|
| "You need one giant model trained on everything to handle vision, speech and text together" | The Training-Free Multimodal Large Language Model Orchestration framework coordinates existing specialised models with a controller LLM and no additional joint training |
| "A protocol like MCP makes a system secure by design" | ForgeFlow's own documentation notes tenant query scoping is still "in progress" — the protocol standardises tool access, it doesn't by itself finish access control |
| "Orchestration is always slower than one model doing everything" | The MLLM Orchestration paper reports a 10.3% latency reduction compared with jointly-trained approaches, not an increase |
| "LangGraph and MCP do the same job" | LangGraph models the workflow as a stateful graph; MCP standardises how agents reach tools — they solve different parts of the problem |
| "Giving agents tool access through a shared protocol means every step is automatic" | ForgeFlow gates high-impact actions behind human approval before execution, regardless of how agents reach tools |
Worked example
A household assistant receives a photo of a broken appliance and a spoken question about whether it's covered under warranty.
- The controller receives both the image and the audio in the same request and decides that the image needs a vision specialist and the audio needs a transcriber, routing each accordingly.
- The vision specialist describes the appliance and visible damage from the photo, using the kind of vision-LLM description ForgeFlow's multimodal module performs.
- The speech specialist transcribes the spoken question into text.
- The controller merges the image description and the transcribed question into one combined request.
- An agent reaches the warranty document only through the shared MCP tool server, retrieving the relevant clause rather than through a bespoke integration built just for this task.
- Shared memory is checked for whether this appliance or warranty has come up in an earlier conversation, using semantic recall rather than an exact-text match.
- The controller proposes an answer — illustrative: "likely covered, serial number required" — and, because this touches a warranty claim rather than just information, the proposed action is held for a person to approve before any claim is actually filed.
- Once approved, the reply goes back to the family, with the routing and tool calls that produced it left visible in the checkpointed graph for later audit.
Should you use it?
Use orchestration when several specialist models already do their individual jobs well and the task is to combine them without retraining anything — and when you need to add or swap a specialist later without rebuilding the whole system. Skip it for a single-modality task where a dispatcher and multiple agents only add latency, and be careful with it for real-time, full-duplex interaction where round trips between agents are hard to hide.
The cheapest first experiment is not building a controller at all: take two specialists you already trust — a transcriber and a document retriever — and wire them through one shared protocol with a single routing decision in between, before adding a third agent.
Say it in the room
"Use it when you already have specialists that work and need coordinating, not retraining; skip it when one model already does the job end to end fast enough."
Lines to carry
- "One supervisor, many specialists, no retraining."
- "Each sense gets its own expert."
- "Checkpoints beat one giant prompt."
- "A shared notebook, not a shared mind."
- "Training-free doesn't mean coordination-free."
- "The protocol standardises tool access; it doesn't finish security on its own."
FAQ
Is this the same as just calling several APIs in sequence?
No. Calling APIs in sequence has no shared state, no memory and no standard way for one step to hand context to the next. Orchestration adds a controller that decides routing, a shared protocol for tool access, and a memory store that later steps can query — the coordination is the point, not the individual calls.
Does orchestration mean the system "understands" images and speech better than one big model?
No model involved understands anything; each specialist predicts outputs from patterns in its training data, and the controller predicts routing decisions the same way. What orchestration changes is which specialist handles which input, and whether results are merged well — not whether anything is understood.
Why not just wait for one really good omni-model instead of wiring specialists together?
Jointly-trained omni-models such as Gander, described as unifying omni perception, realtime interaction and agentic capability in a single framework, do exist and are improving. But they require retraining to add a new skill, whereas an orchestrated system can add a specialist agent behind the same protocol without retraining anything already working.
Is the Model Context Protocol a security feature?
Not a security boundary
MCP standardises how agents reach tools; it does not by itself enforce who is allowed to see what. ForgeFlow's own documentation notes that tenant query scoping is still "in progress" even though its agents already route tool access through MCP — permissions have to be enforced separately.
What breaks first if I try this for a real-time voice assistant?
Latency. Routing a request through a controller, out to a specialist, back for merging, and potentially out again for an approval gate adds round trips that a single model handling full-duplex conversation doesn't have. Careful engineering can claw this back, but it's the first thing to measure before committing to orchestration for anything real-time.
Glossary
- Multi-modal — involving more than one type of input or output, such as text, images, audio or video, in the same system.
- Orchestration — the coordination of multiple separate models, agents or tools to complete a task together, rather than one model doing it alone.
- Supervisor agent / controller LLM — a model whose job is to decide which specialist agent or tool should handle each part of an incoming request.
- MCP (Model Context Protocol) — a standard way for AI agents to access external tools and data so that different agents don't need custom integrations for each tool.
- LangGraph — a framework for building agent workflows as a graph of connected steps (nodes) with shared, trackable state.
- Hub-and-spoke architecture — a design where a central hub (the supervisor) communicates with multiple spokes (specialist agents), rather than agents talking directly to each other.
- Semantic memory / pgvector — a way of storing information as numerical representations (vectors) so a system can retrieve the most relevant past information by similarity rather than exact match.
- Human-in-the-loop approval — a step where a human must approve an AI agent's proposed action before it is carried out, used for high-impact or risky actions.
- Training-free orchestration — combining existing pre-trained models to perform multimodal tasks without further training them jointly on combined data.
- Jointly-trained model — a single model trained from the start on multiple modalities together, as opposed to combining separately trained models afterward.
- Full-duplex interaction — a conversational mode where a system can listen and speak (or be interrupted) at the same time, rather than taking strict turns.
- Vision-LLM — a large language model extended or paired with the ability to interpret images.
- Checkpointed state machine — a workflow execution model that saves its progress at defined points so it can resume or be audited rather than running as one unbroken process.
Sources
- JoelJohnsonThomas/ForgeFlow — GitHub
- Developing Multimodal Human-AI Interaction Systems Using Multi-Agent Frameworks — ACM, Companion Proceedings of the 27th International Conference on Multimodal Interaction
- Training-Free Multimodal Large Language Model Orchestration — arXiv
- docs/ARCHITECTURE.md — GitHub
- Omni Interaction Agent Technical Report — arXiv