DM Explains · No. 010 · Orchestration
Hybrid orchestration layers: one control plane, many disconnected houses
A centralised system can tell hundreds of sites what to do without ever moving their data or losing control when the line drops.
· 14 min read
Read me first
A hybrid orchestration layer genuinely does let one team run infrastructure across cloud, on-premises and edge sites from a single control surface, but only when the sites can also run without it; the control plane commands, it never has to be present for the work to happen.
Illustrative: an 800-store rollout — 1 each store links via Direct Connect and VPN → 2 a cloud engine issues commands, not workloads → 3 EKS Anywhere clusters run entirely on-store hardware → 4 ops sees all 800 through one console → 5 a new checkout service is placed at the edge automatically → 6 central policy is checked against local judgement → 7 the catalogue updates in the cloud while the data never leaves the store → stores keep serving even when disconnected
A retail chain needs to push a software update to bare-metal servers in eight hundred stores. Some of those stores lose internet for hours at a time. A regulator insists that customer data collected in that country never leaves it. One central IT team, eight hundred sites, and no acceptable answer that starts with "we'll just route everything through the cloud."
Get this wrong in either direction and it's expensive. Route control through the cloud and lose the link, and eight hundred stores are suddenly unmanaged. Route customer data through a cloud region to make coordination easier, and the company has broken a law, not just a service-level agreement.
Everyday picture. Amma coordinates the household from the kitchen phone, calling Papa at the office, Dadi in her room and Arjun at school, telling each what needs doing without doing any of it herself. When the phone line to Dadi's room goes dead for an afternoon, Dadi keeps the house running on her own judgement and simply reports back once the line returns.
Where the picture breaks. A phone call assumes someone always answers. A hybrid orchestration layer has to survive the line being dead for hours, and then reconcile exactly what happened while it was down, which a family phone tree never has to prove.
Why it exists now
Running AI and multi-model workloads at scale surfaced a problem that plain multi-cloud tools were never built for: infrastructure that must respect data sovereignty law, tolerate genuinely disconnected networks, and stay within latency limits, all while being managed by a small central team rather than eight hundred local ones.
In 2026, AWS published an architecture pattern for exactly this: serverless services centrally orchestrating thousands of on-premises bare-metal servers running Amazon EKS Anywhere, explicitly built for cases needing the Kubernetes control plane to stay on-premises for sovereignty or for what the industry calls DDIL conditions — disconnected, disrupted, intermittent, limited networks (AWS Architecture Blog, 2026). Around the same time, Dell framed its AI Data Platform around a single orchestration layer spanning cloud, on-premises and edge, with compute provisioned wherever the data already sits (Dell Technologies Info Hub, 2026). Analysts have started treating this as a category in its own right: Gartner's Critical Capabilities for Distributed Hybrid Infrastructure evaluates providers across nine capabilities and four use cases (Gartner, 2026). The concept is also moving toward shared standards, with an IETF draft proposing a unified framework for cross-cloud resource management (IETF Datatracker).
The counterintuitive part: the answer to "how do we manage everything from the cloud" is often "don't run the control plane in the cloud at all." Compute follows the data, not the other way round.
How it works
Connectivity first
No coordination without a secure line home
What it is. Dedicated or encrypted network links join the central control layer to every remote site before any coordination can happen.
Why it exists. State cannot be synchronised, and commands cannot be issued, without a secure path between the orchestrator and the infrastructure it manages.
Where it breaks. The link is assumed reliable enough for control traffic even if it isn't reliable enough for continuous data streaming. If it's down most of the time, the whole model degrades toward manual operation.
How that's handled. Architectures separate control traffic, which is small and infrequent, from data traffic, which is not, so a thin, intermittent connection is often enough to keep coordination alive.
In technical terms
In technical terms: AWS's hybrid cloud orchestration architecture uses AWS Direct Connect for dedicated private connections and AWS Site-to-Site VPN for encrypted tunnels to link AWS services to on-premises infrastructure (AWS Architecture Blog, 2026).
Central engine
One place issues the commands, not the work
What it is. A serverless or managed control plane in the cloud issues lifecycle commands — configure, deploy, upgrade — and tracks the state of every site centrally.
Why it exists. Centralising command issuance lets a small team manage hundreds of geographically dispersed sites without operating each one by hand.
Where it breaks. The engine can command, but it cannot force execution if a site is unreachable. Commands queue rather than complete.
How that's handled. State tracking assumes eventual, not immediate, consistency: a command issued today may not execute until a store's connection returns tomorrow, and the system is designed to know the difference between "done" and "queued."
In technical terms
In technical terms: AWS's architecture consists of three layers — a centralised orchestration engine on AWS, distributed on-premises infrastructure, and hybrid connectivity linking them — coordinated via serverless technologies with an Inventory Management System tracking state across sites (AWS Architecture Blog, 2026).
Local execution
Work happens where the data already lives
What it is. The cluster itself, and often the entire Kubernetes control plane, runs on local hardware regardless of whether the site can reach the cloud.
Why it exists. Data sovereignty rules, regulation, or genuinely unreliable networks make a cloud-hosted control plane a liability rather than a convenience.
Where it breaks. Running a full local control plane means each site can genuinely diverge in state during an outage. That divergence has to be reconciled afterwards, not assumed away.
How that's handled. Distributions built for this case run entirely on customer hardware, distinct from designs that keep the control plane in the cloud and only extend it outward.
In technical terms
In technical terms: Amazon EKS Anywhere runs the entire Kubernetes cluster on the customer's own hardware, recommended for data sovereignty, regulatory, or DDIL network reasons, whereas EKS Hybrid Nodes keeps the control plane in the cloud and is recommended when a managed cloud control plane is acceptable (AWS Architecture Blog, 2026).
One pane of glass
Same policy, different postcodes
What it is. A unified management layer applies orchestration, monitoring and governance consistently across on-premises, multicloud and edge, instead of a separate toolset for each.
Why it exists. Without a shared policy and visibility layer, teams end up running one tool per environment, and inconsistency between those tools becomes the operational risk.
Where it breaks. Treating every site as identical invites bad decisions; a single pane of glass can paper over real differences in latency and capability between a cloud region and a store's back-office server.
How that's handled. Analysts evaluate distributed hybrid infrastructure providers specifically on their ability to handle these differences, not just present a common interface.
In technical terms
In technical terms: IBM defines distributed hybrid infrastructure as spanning on-premises, multicloud and edge computing under a unified management layer providing orchestration, automation, monitoring and governance tools (IBM), and Gartner's Critical Capabilities for Distributed Hybrid Infrastructure evaluates provider offerings across nine capabilities and four use cases (Gartner, 2026).
Service placement
Compute goes where it's needed, automatically
What it is. A service orchestration layer decides where application components actually run, across multiple cloud and edge providers at once.
Why it exists. Multi-provider, multi-location workloads need automated placement rather than manual scheduling to optimise for latency, cost and available resources.
Where it breaks. Automated placement optimises for the metrics it's fed. If latency or cost data is stale, the placement decision goes quietly wrong, and nobody notices until performance suffers.
How that's handled. The service orchestrator hands infrastructure-level decisions to a separate multi-cloud orchestrator, keeping the "what should run where" question distinct from the "how do we provision that" question.
In technical terms
In technical terms: The IPCEI-CIS reference architecture defines a Service Orchestration Layer that manages multiple cloud and edge services and applications in a unified, automated way across diverse multi-provider environments, and can communicate with a Multi-Cloud Orchestrator managing the virtualised infrastructure layer beneath it (IPCEI-CIS Reference Architecture, 2026).
Federated governance
Rules are central; judgement stays local
What it is. Policy is defined once at a central hub, while far-edge nodes keep the authority to act on their own during disconnection.
Why it exists. A fully centralised model fails the moment edge connectivity becomes unreliable, so governance and execution are deliberately separated.
Where it breaks. Local autonomy means local decisions can briefly conflict with central policy. That gap has to be resolved explicitly when the site reconnects, not assumed to close itself.
How that's handled. A hub cluster owns policy; edge nodes execute and adapt independently, then reconcile.
In technical terms
In technical terms: The CODECO approach defines a hybrid governance model for federated container orchestration that centralises governance at a hub cluster using Open Cluster Management while allowing decentralised execution and learning at the far edge (CODECO, arXiv, 2026).
Metadata split
The catalogue travels; the books stay put
What it is. Metadata, lineage and policy are governed centrally, while the actual data and models never leave the site that produced them.
Why it exists. This satisfies data residency requirements while still giving central teams visibility into what exists and where, without moving the regulated content itself.
Where it breaks. This only holds if the metadata genuinely contains no regulated content — a check teams sometimes skip.
How that's handled. Compute is provisioned wherever the data already sits, rather than the data being moved to wherever compute is cheapest.
In technical terms
In technical terms: Dell's AI Data Platform design keeps metadata, lineage and policy governed centrally in the cloud while data and models remain on-premises, provisioning compute in cloud, hybrid, or fully on-premises environments so that compute follows the data (Dell Technologies Info Hub, 2026).
Compared with what came before
| Per-environment tooling | Hybrid orchestration layer | |
|---|---|---|
| Control surface | One tool per cloud, one per on-premises stack | One control plane across all sites |
| Behaviour offline | Site is unmanaged until reconnected | Site keeps executing locally, reconciles later |
| Data movement | Often centralised for visibility | Metadata travels; data and models stay put |
| Governance | Ad hoc, per team | Policy defined once, enforced everywhere |
| What you give up | Simplicity of a single environment | Consistency assumptions; eventual, not instant, state |
The judgement: a hybrid orchestration layer trades the comfort of one simple environment for the ability to manage many genuinely different ones without losing control the moment one of them goes quiet. That trade is only worth making once the number of sites, or the sovereignty constraints on them, make per-environment tooling unmanageable in the first place.
Where it breaks
Illustrative: 800-store rollout — 1 policy pushed to every EKS Anywhere cluster → 2 each store checked over Direct Connect/VPN → connected store 3 executes immediately → disconnected store 4 queues locally, still serving customers → 5 reconciles once reconnected → fleet converged
A hybrid orchestration layer holds up well when disconnection is expected and designed for, but it strains the moment teams pretend the network is more reliable than it is.
Not a real-time control system
The control plane issues commands and tracks state, but during a genuine outage a site is on its own. If the design assumes the line rarely drops, an outage becomes an unmanaged period rather than a handled one.
Reconciliation is the hard part, not the theory
Local autonomy during disconnection is only safe if there's an explicit process for resolving conflicts once a site reconnects. The CODECO work exists precisely because purely centralised models fail under variable connectivity (CODECO, arXiv, 2026); a design without an equivalent reconciliation step is quietly fragile.
A single pane of glass can hide real differences
Presenting cloud, on-premises and edge sites through one interface makes them look interchangeable. They rarely are: a store's back-office server and a cloud region do not have the same latency, capacity or failure behaviour, and placement decisions that ignore this go wrong silently.
Numbers to carry
| Figure | What it describes |
|---|---|
| 2026 | Year AWS published its hybrid cloud orchestration architecture pattern for on-premises bare-metal fleets (AWS Architecture Blog) |
| Nine capabilities, four use cases | Scope of Gartner's Critical Capabities for Distributed Hybrid Infrastructure evaluation, as of 2026 (Gartner) |
| Seven-stop spectrum | Breadth of infrastructure Spectro Cloud's hybrid AI control plane is described as coordinating (Spectro Cloud) |
| 2026 | Year Dell published its AI Data Platform framing around compute following data (Dell Technologies Info Hub) |
| 2026 | Year the CODECO federated container orchestration paper describing hub-and-far-edge governance was published (CODECO, arXiv) |
Myths worth killing
| Commonly repeated | What is actually true |
|---|---|
| "Hybrid orchestration means the cloud reaches out and runs everything" | Designs like EKS Anywhere deliberately keep the whole Kubernetes control plane on customer hardware; the cloud issues commands, it does not run the cluster |
| "If the network drops, the site is down" | Federated governance models are built specifically so far-edge nodes keep executing and adapting during disconnection, then reconcile on reconnect |
| "One pane of glass means every site is managed identically" | The unified layer applies consistent policy and monitoring, but placement and governance decisions still have to account for each site's real latency and capability differences |
| "This is just multi-cloud management" | Multi-cloud tools coordinate across public clouds only; hybrid orchestration additionally spans on-premises and edge sites, including disconnected ones |
| "Centralising means centralising the data too" | Several designs centralise only metadata, lineage and policy, keeping data and models on-premises throughout |
Worked example
The 800-store rollout, illustrative in its specific figures:
- Central IT issues a single upgrade command from the cloud-based orchestration engine, targeting all eight hundred stores at once.
- The Inventory Management System records which stores have acknowledged the command and which have not, based on AWS's pattern of tracking state across dispersed sites.
- Store A, connected throughout, receives the command over its Site-to-Site VPN link, applies the upgrade to its local EKS Anywhere cluster, and reports success within the hour — illustrative timing, not a measured figure.
- Store B loses its connection partway through the rollout. Its local cluster keeps serving customers on the old version; the command sits queued centrally rather than failing outright.
- Store B's local system, once reconnected hours later, pulls the queued command, applies the upgrade, and reports its new state back to the Inventory Management System.
- The regulator's requirement is met throughout because customer transaction data at every store never left the country of collection; only commands and state metadata travelled to and from the cloud.
- Central IT ends the day with a dashboard showing eight hundred stores, most upgraded within hours, a handful queued and completed later, none managed by hand.
Should you use it?
Use a hybrid orchestration layer when infrastructure is genuinely dispersed across many sites that need central visibility but local execution, when data sovereignty or regulation requires data and control to stay on-premises, or when networks to those sites are disconnected, disrupted, intermittent or limited by nature rather than by accident. Skip it when workloads sit inside a single cloud provider with no on-premises or edge component, or when the scale is small enough that a single manually-managed cluster is genuinely simpler than building reconciliation logic for outages that will rarely happen.
The cheapest first experiment is not a rollout across eight hundred sites; it is standing up the connectivity and central engine for a handful of representative sites — one always-connected, one deliberately disconnected for a test window — and checking that the reconciliation step actually produces the state you expect once it reconnects.
Say it in the room
"Use it when sites must be centrally visible but locally sovereign and sometimes disconnected; skip it when everything already lives in one cloud with one reliable network."
Lines to carry
- "Compute follows the data, not the other way round."
- "One control plane, many houses that can lock their own doors."
- "Central policy, local judgement."
- "Coordination, not custody."
- "The command queues; it doesn't fail, if the design expects the line to drop."
- "A single pane of glass can hide real differences between sites."
FAQ
Is this the same as multi-cloud management?
No. Multi-cloud tools coordinate workloads across several public cloud providers only. A hybrid orchestration layer also spans on-premises data centres and edge sites, including ones that go offline for hours at a time, which most multi-cloud tools were never built to tolerate.
Does the control plane have to run in the cloud?
Not necessarily, and often deliberately not. Designs like Amazon EKS Anywhere keep the entire Kubernetes control plane on the customer's own hardware specifically so sovereignty rules or unreliable networks don't force a dependency on the cloud. Other designs, like EKS Hybrid Nodes, do keep the control plane in the cloud when that's acceptable.
What happens to commands issued while a site is offline?
They queue. The centralised orchestration engine tracks state through an inventory system and reissues or resumes commands once the site reconnects, rather than treating the outage as a failure.
Where does customer data actually go in a design like this?
In several architectures, only metadata, lineage and policy travel to the central layer; the data and models themselves stay on-premises. Compute is provisioned wherever the data already is, rather than the data being moved to meet the compute.
Is this overkill for a smaller company?
Usually, yes. If your infrastructure fits in one cloud provider with no on-premises or edge component, or if a single team can manage everything manually without strain, the reconciliation and governance machinery this pattern requires is unnecessary cost.
Glossary
- Orchestration layer — software that coordinates and automates the deployment, scaling and lifecycle of infrastructure or workloads across multiple systems.
- Control plane — the part of a system that decides where and how workloads run, as opposed to the data plane that actually processes them.
- Edge computing — running computation close to where data is generated, rather than in a distant data centre.
- DDIL (disconnected, disrupted, intermittent, limited) — a description of network conditions where connectivity to a central cloud cannot be relied upon.
- Bare-metal server — a physical server dedicated to one user, not divided by virtualisation software.
- Kubernetes — an open-source system for automating deployment, scaling and management of containerised applications.
- EKS Anywhere — Amazon's Kubernetes distribution that customers can run entirely on their own hardware.
- Federation (in orchestration) — connecting multiple separately-managed clusters so they can be coordinated as a group while retaining local control.
- Data sovereignty — the requirement that data be stored and processed within specific legal or geographic boundaries.
- Multi-cloud — using more than one public cloud provider for different parts of an organisation's infrastructure.
- Hybrid cloud — a mix of on-premises infrastructure and one or more clouds, used together as one system.
- Inventory Management System — a component that tracks the current state and configuration of every managed site in a distributed system.
- Service orchestration — automated coordination of how application components are deployed and connected across cloud and edge providers.
- Open Cluster Management (OCM) — a Kubernetes-native framework for registering, managing and applying policy across multiple clusters.
- Compute-follows-data — a design principle where processing moves to wherever data already resides, instead of moving the data.
Sources
- Hybrid cloud orchestration: Modernizing on-premises infrastructure management with AWS — AWS Architecture Blog, 2026
- What is Distributed Hybrid Infrastructure? — IBM
- Critical Capabilities for Distributed Hybrid Infrastructure — Gartner, 2026
- IPCEI-CIS Reference Architecture — OpenNebula / IPCEI-CIS, 2026
- The control plane for hybrid AI infrastructure (part 2) — Spectro Cloud
- draft-zhou-crosscloud-orchestration-02 — IETF Datatracker
- From Hybrid Data to AI-Ready Workflows — Dell Technologies Info Hub, 2026
- Towards Scalable Federated Container Orchestration: The CODECO Approach — arXiv, 2026