DM Explains · No. 037 · Orchestration
Hybrid orchestration: why AI agents need a boss, not a dictator
Enterprises running multi-agent AI are discovering that total central control chokes scale, and total freedom kills governance — so the winning pattern gives agents local autonomy inside boundaries a controller still enforces.
· 13 min read
Hybrid orchestration: why AI agents need a boss, not a dictator
Read me first
Hybrid orchestration genuinely does fix the multi-agent bottleneck, but only when the controller stops approving every step and starts guarding every stage; agents talk freely inside the fence, the boss just locks the gates.
Illustrative: a loan-approval workflow splits spec from execution → orchestrator sets a compliance stage and a research stage → agents classify, fetch documents and draft a memo by talking directly → optimiser maps the drafting step to GPU vs edge, informed by systems reducing GPU usage up to 2.8x and cost up to 4.3x → an eight-node Jetson Nano cluster keeps classification running when the network drops → gateway checks the agent's intent before it calls the document store → memo is approved and released
A company wires up a dozen AI agents to handle a customer request end to end — classify, retrieve, draft, approve, send — and routes every single handoff through one central controller that approves each micro-step. It feels careful. It behaves like a committee that signs off on every word of every email: the controller becomes a bottleneck, latency balloons, GPU bills climb, and the one thing the company actually needed — an audit trail showing nothing unsafe happened — barely improves, because the controller is too busy micromanaging to meaningfully enforce policy.
Everyday picture. Amma sets the week's meal plan and the shopping budget, like a central orchestrator. But once Arjun and Meera are in the kitchen cooking dinner for that one stage, they coordinate directly — who chops, who stirs — without checking in with her on every move. Bruno is allowed to roam the garden freely, as long as he doesn't cross into the kitchen.
Where the picture breaks. Amma can always walk in and physically override anything. A software orchestrator's boundary is a programmed check, not an ever-present supervisor, so a gap in that check lets unsafe actions slip through unnoticed.
Why it exists now
Through 2025 and 2026 several concrete pieces converged. NVIDIA's GTC San Jose 2025 session showed an eight-node Jetson Nano cluster running multiple LLMs with cloud GPU failover, proving edge devices can carry real inference load, not just cached replies. The USENIX OSDI "Murakkab" paper presented a profile-guided runtime for serving agentic workflows with measured efficiency gains. An ACM workshop proposed a declarative composition model for heterogeneous agents, separating what a workflow should do from who runs it. And industry analysis increasingly frames hybrid orchestration — centralised governance paired with decentralised choreography — as the default enterprise pattern for agentic AI deployments.
The stakes are not academic. Enterprises moved from single-model chatbots to multi-agent, multi-modal systems that must run reliably, cheaply and safely across unpredictable environments. Pure central control doesn't scale. Pure peer-to-peer coordination isn't governable. Both failure modes show up in production, and the fix that's converging across research and vendor reports is the same: stop treating every agent handshake as a decision the controller must make, and start treating the controller as something that only checks the doors between rooms.
How it works
Split spec from execution
Decide what happens before deciding who does it
What it is. The workflow's steps and dependencies are described separately from which model, provider or machine will run them.
Why it exists. Binding a workflow to a specific model or piece of hardware at design time locks in choices that will be wrong later, when prices, context limits or available GPUs change. A declarative specification lets a runtime reason about cost, latency and accuracy trade-offs after the fact.
Where it breaks. If the declarative layer is too vague about constraints, the runtime can make technically valid but practically unacceptable choices — correct on paper, wrong in production.
How that's handled. The specification carries explicit constraints (budget ceilings, latency targets, data residency rules), not just a task list.
In technical terms
In technical terms: a declarative composition model separates agent declaration, dispatch and execution, shifting integration logic from imperative orchestration code into declarative specifications.
Orchestrator sets stages
One controller owns structure, not every step
What it is. A central service defines the workflow's stages and checks boundaries at transitions between them, not inside them.
Why it exists. Centralised control gives the auditability and compliance that regulators and risk teams actually ask for, without forcing the controller to be present for every internal exchange.
Where it breaks. Badly drawn stage boundaries either let risky behaviour through mid-stage, or recreate the micromanagement bottleneck by making the stages too small to matter.
How that's handled. Stages are drawn around compliance-sensitive transitions — document retrieval finishing before decision drafting begins — rather than around every message.
In technical terms
In technical terms: most enterprise agentic AI deployments combine centralised orchestration for structured workflows with choreography for dynamic interactions within those workflows.
Agents choreograph
Inside the fence, agents just talk to each other
What it is. Within a stage, agents pass messages and delegate sub-tasks to each other directly, without asking the orchestrator's permission at each turn.
Why it exists. Removing the orchestrator as a per-step bottleneck is what makes fast, dynamic interaction possible at all; a controller in the loop on every exchange cannot keep up with agents working at model speed.
Where it breaks. Agents have overlapping roles and loosely defined capabilities, unlike deterministic microservices, so choreography can quietly produce goal misalignment that nobody notices until the stage ends.
How that's handled. The stage has a clear exit check — the orchestrator inspects the stage's output against its contract before letting the next stage begin, rather than trusting that internal coordination went well.
In technical terms
In technical terms: a comparative study of hybrid microservice and LLM-based multi-agent architectures analyses orchestration and API design across eight architectural dimensions, flagging that overlapping agent roles need handling microservices never required.
Optimiser maps resources
Measure first, then decide where things run
What it is. A profile-guided runtime assigns workflow components to specific models and hardware to meet cost, latency or accuracy targets.
Why it exists. Treating a workflow as an opaque sequence of calls tightly bound to fixed models wastes resources; profiling what each component actually does lets the runtime place work where it's cheapest to run well.
Where it breaks. Profiling assumes workloads are stable enough to measure; highly novel or one-off tasks give the optimiser little to learn from.
How that's handled. The optimiser is paired with explicit service-level objectives, so it has a target to optimise against rather than guessing at "good enough".
In technical terms
In technical terms: Murakkab, a resource-efficient serving system for agentic workflows evaluated at USENIX OSDI, cut GPU usage by up to 2.8×, energy by up to 3.7× and cost by up to 4.3× compared with prior serving systems, while still meeting user-defined service-level objectives.
Edge-cloud failover
The edge keeps working when the network doesn't
What it is. Local edge devices keep serving models during outages, with cloud GPUs folded back in once connectivity returns.
Why it exists. A hybrid deployment spanning edge and cloud has to survive the edge losing its connection, which happens constantly in field, retail and air-gapped settings.
Where it breaks. Edge hardware has real limits on model size and throughput, so failover buys resilience, not equivalent capability.
How that's handled. Smaller models are pre-placed on edge nodes specifically for the failure case, while larger models stay cloud-side for when the link is up.
In technical terms
In technical terms: an eight-node NVIDIA Jetson Nano cluster served multiple LLMs — Gemma 2 2B and Llama 3.2 3B — maintaining continuous operation in air-gapped environments and during network disruption, while integrating cloud GPUs for additional scale.
Gateway checks intent
Ask before the tool call, not after the fact
What it is. A runtime gateway evaluates what an agent intends to do and can block a tool call before it executes, rather than relying only on logs reviewed afterward.
Why it exists. As agents gain real operational reach over enterprise systems — sending money, editing records, calling external APIs — a log that says what already happened is too late to matter.
Where it breaks. Intent evaluation is itself a judgement made by another model-based system; it is a reduction of risk, not a guarantee.
How that's handled. The gateway sits as a mandatory checkpoint between agent and tool, so even an agent that reasons its way around policy still has to pass through the check physically, not just logically.
In technical terms
In technical terms: zero-trust runtime gateways evaluate agent intent and mediate tool calls before execution, replacing reliance on static, post-hoc audit logs.
Compared with what came before
Illustrative: a loan agent classifies the application, pulls title and income docs under a fenced retrieval stage, drafts a memo with peer agents researching freely, hits a boundary check before approval, and the decision is issued — auditable at the edges, fast in the middle.
| Fully centralised orchestration | Pure decentralised choreography | Hybrid orchestration | |
|---|---|---|---|
| Who approves each step | The controller, every time | No one; agents self-coordinate | Controller at stage boundaries only |
| Governance and audit | Strong, but slow | Weak or absent | Enforced at the points that matter |
| Speed inside a task | Bottlenecked by the controller | Fast | Fast within a stage |
| Failure mode | Latency and cost blow-up | Untracked goal drift | Depends on where stage lines are drawn |
The judgement here is unglamorous: nobody actually wants either extreme, they just end up there by accident — centralised because "safety first" sounded simpler to build, decentralised because agent frameworks made peer coordination the path of least resistance. Hybrid orchestration is what's left once a team notices the bottleneck or the audit gap and has to fix it without throwing away the other half.
Where it breaks
Hybrid orchestration helps when the stage boundaries are drawn correctly and the gateway's intent checks are trustworthy; it does not help, and can actively mislead, when either of those is weak.
Stage boundaries are a design choice, not a law
Nothing forces a team to draw stage boundaries around the right risk points. A boundary placed too late in a compliance-sensitive workflow lets agents choreograph their way through exactly the step that needed a controller watching. This isn't a flaw in the pattern; it's a flaw in how it was applied.
Multi-agent systems also inherit a problem microservices never had: agents have overlapping roles and capabilities that are not precisely defined, so even within a correctly bounded stage, goal misalignment between agents can go unnoticed until the stage's output is checked — which is only as good as that exit check. An ICSE 2026 workshop paper on automated orchestration for LLM-based multi-agent systems names scalability, robustness and the operational discipline of running agents in production ("AgentOps") as the open challenges here, not solved problems.
Cost is a real variable too: a profile-guided optimiser only pays off if workloads are stable enough to profile. A one-off, highly novel task gives the optimiser nothing to learn from, and the system defaults to whatever placement is safest rather than cheapest.
Numbers to carry
| Figure | Value |
|---|---|
| GPU usage reduction, Murakkab vs prior serving systems | up to 2.8× |
| Energy reduction, Murakkab vs prior serving systems | up to 3.7× |
| Cost reduction, Murakkab vs prior serving systems | up to 4.3× |
| Correctness gain, A.DOT Planner vs baseline | 14.8% absolute |
| Completeness gain, A.DOT Planner vs baseline | 10.7% absolute |
| Edge cluster demonstrated at NVIDIA GTC San Jose 2025 | eight-node Jetson Nano cluster |
| Architectural dimensions compared, hybrid microservice/LLM-MAS study | eight |
Myths worth killing
| Commonly repeated | What is actually true |
|---|---|
| "More autonomy for agents always means less control." | Hybrid systems show agents can coordinate freely inside a stage while the controller still enforces governance at the boundary. |
| "Central orchestration is the safe choice." | Centralised orchestration that checks every step is slow, expensive and doesn't necessarily improve safety, because the controller is too busy to enforce policy meaningfully. |
| "Hybrid orchestration is just microservices with LLMs." | LLM-based agents have overlapping, loosely defined roles that microservices never had, so the pattern borrows modularity but still has to manage goal misalignment directly. |
| "RAG already solves multi-step reasoning." | Brute-force retrieval-augmented generation merges results from each store after the fact and lacks explicit support for multi-hop reasoning, which is why DAG-based planners exist. |
| "Edge devices can only serve cached answers during an outage." | A demonstrated eight-node Jetson Nano cluster ran full inference locally across multiple LLMs during network disruption, not just replayed cached responses. |
Worked example
- The loan-approval workflow starts. An applicant submits a loan request; the declarative specification lists the stages — classify, retrieve documents, draft decision memo, approve — without yet saying which model or machine will run each one.
- The orchestrator opens the retrieval stage. It is marked compliance-sensitive, so the orchestrator checks entry conditions (applicant identity verified, documents available) before letting any agent proceed.
- Agents choreograph inside retrieval. A document-fetching agent and a classification agent exchange results directly — pulling from multiple document stores — without asking the orchestrator's permission at each individual fetch.
- The orchestrator checks the exit. Before drafting begins, it confirms retrieval actually finished and the documents meet policy requirements; only then does the next stage open.
- The optimiser places the drafting stage. Illustrative: based on profiling, it routes the memo-drafting task to a larger cloud model for accuracy and the document-summarising sub-task to a smaller, cheaper model, aiming to hit a target latency and cost ceiling it was given.
- The gateway checks the approval call. Before the decision-memo agent can write the final approval into the loan system, the zero-trust gateway evaluates the intent of that specific call and either lets it through or blocks it for human review.
- The network drops mid-process. Illustrative: if the branch office loses connectivity, the edge node keeps serving the smaller model locally to avoid stalling the applicant entirely, and reconciles with cloud resources once the link returns.
Should you use it?
Hybrid orchestration earns its complexity when a workflow genuinely mixes structured, auditable stages with fast, exploratory agent-to-agent work, and when the deployment has to survive network gaps between edge and cloud. It is overkill for a single-model, single-step application, and it is the wrong fix when no governance is actually needed — in that case, pure choreography is simpler and nothing is lost by skipping the controller.
The cheapest first experiment is not infrastructure, it's a boundary drawing exercise: take one existing agent workflow, mark the two or three points where governance genuinely needs to check something, and remove per-step approval from everywhere else. Measure latency and cost before and after; the gain shows up without touching a model.
Say it in the room
"Don't route every agent handoff through a controller; draw the stage boundaries where governance actually matters, and let agents talk freely inside them."
Lines to carry
- "Control the boundary, not the step."
- "Autonomy inside a fence beats permission for every move."
- "Govern the stage transitions; let the stage sort itself out."
- "Measure before you place work, not after."
- "The edge keeps thinking when the line goes dead."
FAQ
Isn't this just microservices with extra steps?
Not quite. Microservices have deterministic, well-defined interfaces. Agents built on language models have overlapping roles and capabilities that are not precisely specified, so a hybrid architecture borrows the modularity of microservices but still has to manage the goal misalignment that comes from agents whose boundaries aren't crisp.
Does choreography mean agents can do anything inside a stage?
Within the stage's declared boundary, yes, agents coordinate directly without asking the orchestrator at every turn. But they cannot exceed that boundary without the orchestrator's approval, and the orchestrator checks the stage's output against its contract before the next stage opens.
How is this different from a RAG pipeline with multiple retrievers?
Simple retrieval-augmented generation pulls from each data store and merges results after the fact, with no explicit multi-hop reasoning. Orchestrated planners instead decompose a query into dependent sub-steps executed under a coordinated plan, which is why one benchmark saw a 14.8% absolute correctness gain over baseline retrieval approaches.
What happens if the orchestrator itself goes down?
The dossier doesn't cover orchestrator failover directly, but the edge-cloud resilience pattern is the closest analogue: edge nodes keep serving locally during network disruption and reconcile with cloud resources once available. A production hybrid design would need an equivalent answer for the orchestrator itself, which is part of what the ICSE 2026 workshop on automated orchestration lists as an open robustness challenge.
Does this replace human approval for sensitive actions?
No. The zero-trust gateway evaluates intent before a tool call executes, which reduces risk, but it is itself a model-based judgement, not a guarantee. Sensitive actions — like the loan approval write-back — still route through the gateway as a mandatory check, not as a replacement for human sign-off where that's required.
Glossary
- Orchestration — the coordination of multiple AI agents, models or services so a complex task is completed in the right order with the right rules.
- Choreography — a pattern where agents coordinate directly with each other without a central controller directing every step.
- Agentic workflow — a sequence of steps carried out by AI agents that can plan, call tools and make decisions, not just answer a single prompt.
- Service-level objective (SLO) — a target for how a system should perform, such as maximum latency or minimum accuracy.
- Multi-modal agent — an AI agent that can process more than one type of input, such as text, images and sensor data, together.
- Edge-cloud hybrid — a system where some processing happens on local, resource-limited devices and some happens on more powerful cloud servers.
- Declarative composition — describing what a system should do rather than writing step-by-step code for how to do it.
- Zero-trust gateway — a checkpoint that verifies every action's intent before allowing it, rather than assuming anything inside the system is safe.
- DAG (directed acyclic graph) — a diagram of steps connected by arrows, with dependencies but no loops, used to plan execution order.
- Multi-hop reasoning — answering a question by chaining several steps of retrieval or inference rather than a single lookup.
- Profile-guided optimiser — a software component that uses measurements of how a workload behaves to decide how to allocate resources to it.
- Air-gapped environment — a system physically or logically isolated from external networks, including the internet.
- LLM-based multi-agent system (LLM-MAS) — a system of multiple AI agents, each powered by a large language model, working together on a task.
Sources
- Resilient Edge-Cloud Hybrid AI Infrastructure: Orchestrating Multi-Modal Agents in Resource-Constrained Environments — NVIDIA, 2025
- Enterprise Agentic AI Platform Strategy and Multi-Agent Orchestration — COMPEL Framework
- Enterprise Hybrid AI Architecture: 5 Layers Explained — Stonebranch
- Towards Declarative Composition of Heterogeneous AI Agents — ACM
- Agentic DAG-Orchestrated Planner Framework for Multi-Modal, Multi-Hop Question Answering in Hybrid Data Lakes — arXiv
- Murakkab — USENIX OSDI / ACM Digital Library
- A comparative study towards designing a hybrid architecture of microservices and LLM-based multi-agent systems — Queen's University Belfast
- Automated Orchestration for LLM-Based Multi-Agent Systems: Challenges and Opportunities — ICSE 2026 / JAWs 2026