DM Explains · No. 048 · Sovereign AI
Federated inference: running one model across machines that don't trust each other
Splitting a model's computation across edge devices or sovereign clouds so no single node ever sees the full prompt, the full weights, or the full answer.
· 14 min read
Read me first
Federated inference genuinely does let a hospital, a bank and a government ministry share a model without sharing their data — but only when the computation itself is split and blurred, not just routed to a trusted vendor's single cloud.
Illustrative: three hospital sites query the model for a discharge summary — 1 the orchestrator splits layers by each site's compute profile → 2 each site runs its shard on local patient notes only → 3 activations leave each site with calibrated DP noise added → 4 the coordinator combines the three noised fragments into one representation → 5 the merged result is served from enterprise infrastructure, never an external API → 6 a federated scheduler balances load across the three clusters next round
A hospital, a bank and a government ministry each want to use a large model to draft discharge summaries, flag suspicious transactions and summarise policy documents. None of them is allowed to send the raw material — patient notes, account data, classified text — to someone else's servers. The usual answer is to shrink the ambition: run a small model locally and live with worse answers. Federated inference asks a different question: what if the big model's computation itself gets cut into pieces, with no single piece large enough to expose anything on its own?
Get this wrong and the choices are both bad. Send everything to one external API and you've handed over the data you were forbidden to share. Keep everything on one underpowered local machine and you've handed users a model too weak to be useful. Federated inference is the attempt to avoid both trade-offs by making privacy a property of the architecture, not a result of isolation.
Everyday picture. Instead of sending her full diary to a neighbour for advice, Amma tears out only the relevant page, blurs the names on it, and lets the neighbour read just that blurred page before answering. Nobody outside the house ever sees the whole diary.
Where the picture breaks. Federated inference splits a running computation across machines and adds noise to numeric activations — not literal torn-out, redacted pages of text. The diary never actually gets cut up; what gets cut up is a mathematical process, and what gets blurred is a vector of numbers, not a name on a page.
Why it exists now
The problem is old — regulated industries have never been comfortable sending sensitive prompts to a third party's cloud — but through 2025 the architecture for solving it without crippling the model got formalised and productised. An ACM SIGCOMM 2025 poster, published 10 September 2025, introduced "capability-aware model partitioning" across edge devices with differential-privacy noise injected into intermediate activations before they ever leave a device (ACM SIGCOMM 2025). An arXiv position paper from 2025 went further and tried to pin down the first formal definition of federated inference, distinct from federated learning, specifying exactly which things are not allowed to be revealed in plaintext: the client's input, the model's parameters, and the intermediate representations and predictions passed between participants (arXiv, 2025). And vendors moved from research demo to sales pitch: Lenovo, IBM and NVIDIA jointly published a "Sovereign AI Inference" solution brief describing production stacks already in use across financial services, healthcare, government and legal sectors (Lenovo Press).
What changed is not the need — it's that the pieces now exist to build this without hand-waving: formal confidentiality definitions, working partitioning demos, and production serving stacks that enterprises can actually deploy inside their own boundary.
How it works
Model partitioning
Each device does only the part it can carry
What it is. A central orchestrator splits a model's layers across several devices according to how much compute and memory each one has, rather than splitting evenly.
Why it exists. A weak edge device cannot run a full large model by itself; splitting the workload lets it do only the slice it can carry while the system as a whole still produces one answer.
Where it breaks. Partitioning adds coordination overhead between devices. If the network linking them is slow, a split model can end up slower than simply running a smaller model on one machine.
How that's handled. The orchestrator profiles each device's capability before assigning layer ranges, so a weak node gets a small slice and a stronger node carries more.
In technical terms
In technical terms: the Federated Inference (FI) framework presented at ACM SIGCOMM 2025 uses a central orchestrator that adaptively splits models based on individual client device profiles, a technique it terms capability-aware model partitioning (ACM SIGCOMM 2025).
Local computation
Raw data never leaves the room it was born in
What it is. Each node processes its own slice of input where the input lives, and never transmits the raw plaintext input to any other node.
Why it exists. This is the baseline requirement that separates federated inference from simply calling a remote API: the client's input must not be revealed in plaintext to other participants.
Where it breaks. Keeping raw input local says nothing about what happens next — the intermediate values a model computes from that input can still leak information about it.
How that's handled. The intermediate values get protected before they're sent anywhere, which is the next moving part.
In technical terms
In technical terms: a 2025 position paper defines input confidentiality as one of federated inference's formal requirements, alongside the requirement that intermediate representations and per-model predictions must not be observable in plaintext (arXiv, 2025).
Noised transmission
Blur before you share, never share then blur
What it is. Before intermediate activations leave a device, calibrated noise gets injected into them.
Why it exists. Activations can be used to reconstruct the original input they came from, so sending them unprotected would undo the privacy gained by keeping raw data local.
Where it breaks. Too much noise degrades the final answer. How much noise is enough to protect the input but not so much that the model's output becomes useless is a genuinely fiddly, unresolved calibration problem.
How that's handled. The SIGCOMM 2025 framework trades a small accuracy cost for a measurable privacy guarantee by tuning the noise level against task performance.
In technical terms
In technical terms: the FI framework injects calibrated differential privacy noise into intermediate activations before transmission between edge devices (ACM SIGCOMM 2025).
Cross-node combination
Separate owners, one shared answer
What it is. Outputs, partial predictions, or cached intermediate state from independently owned models get combined into one final answer, without any party exposing its full weights to the others.
Why it exists. Collaboration across heterogeneous, independently trained models can produce a better answer than any single constrained model could manage alone — useful when the hospital's model, the lab's model and a specialist model each hold a different slice of competence.
Where it breaks. A 2025 position paper is explicit that ensemble-based collaboration and incentive alignment across different owners remain open friction points, not solved engineering — federated inference cannot simply inherit behaviours from training-time federation or classical ensembles.
How that's handled. FedRefine, a proposed framework, lets heterogeneous on-device large language models collaborate by communicating key-value caches rather than raw parameters, which preserves each model's confidentiality while still improving answer quality.
In technical terms
In technical terms: FedRefine enables heterogeneous on-device LLMs to collaboratively perform inference by communicating KV caches in a privacy-preserving manner (arXiv, 2025), while the same position paper that defines model confidentiality also flags incentive alignment as unresolved (arXiv, 2025).
Boundary serving
The data stays inside the fence that matters
Illustrative: three hospital sites ask for discharge summaries → 1 trust scope found single-org for site A, multi-party for the network query → 2 site A's own data stays on enterprise infrastructure with no external endpoint (Lenovo Sovereign AI Inference) → 3 the three-site query is split across untrusted nodes → 4 the model is partitioned per site's capability profile, capability-aware partitioning → 5 intermediate activations get calibrated differential-privacy noise before transmission, and KV caches are exchanged as in FedRefine → 6 an audit confirms no plaintext leaked → a bounded discharge summary is released
What it is. A production serving stack keeps prompts, retrieved context, the key-value cache, and outputs inside enterprise- or nation-owned infrastructure, with no external inference endpoint required in the path.
Why it exists. Regulated industries need a way to use large models while keeping everything auditable and inside their own jurisdiction, not just trusting a vendor's promise.
Where it breaks. This secures the perimeter of one organisation. It is not the same as the multi-party privacy guarantee that noised, partitioned inference provides across several mutually distrustful nodes — one trusted wall is not the same as several nodes that don't trust each other at all.
How that's handled. By keeping these two problems conceptually separate: boundary serving answers "does data leave our organisation", federated partitioning answers "does any single machine, including ours, see the whole thing".
In technical terms
In technical terms: the Lenovo Sovereign AI Inference architecture, built with IBM and NVIDIA, runs vLLM, NVIDIA Dynamo, LMCache, NIXL/UCX transport and LiteLLM on enterprise-managed Kubernetes, tiering the KV cache to shared storage so nothing in the inference data path has to cross the organisation's boundary (Lenovo Press).
Federated scheduling
Pooling clusters instead of pooling trust
What it is. A toolkit routes inference requests across geographically distributed high-performance computing clusters through a single, familiar interface, keeping some nodes "hot" to cut latency.
Why it exists. Scientific institutions want cloud-like access to large models without depending on a commercial cloud, while still pooling compute that no single site owns in full.
Where it breaks. This solves compute pooling and access, not the harder confidentiality problem of hiding inputs and activations from the clusters doing the computing — pooling clusters is not the same as pooling trust.
How that's handled. The FIRST toolkit reportedly lets researchers generate billions of tokens daily on-premises across federated clusters, using Globus Auth and Globus Compute for authentication and scheduling through an OpenAI-compliant interface.
In technical terms
In technical terms: the FIRST toolkit provides federated inference resource scheduling for scientific AI model access, enabling Inference-as-a-Service across distributed clusters without commercial cloud dependence (arXiv, 2025).
Compared with what came before
| Single-cloud API | On-device-only model | Federated inference | |
|---|---|---|---|
| Who sees the raw input | The API provider | Nobody but the device | No single node alone |
| Model capability available | Full, large model | Limited by one device | Large model, distributed |
| Privacy mechanism | Contractual trust | Physical isolation | Partitioning plus noise |
| Coordination overhead | None | None | Network and orchestration cost |
| Multi-owner collaboration | Not applicable | Not applicable | Supported (e.g. KV-cache sharing) |
The judgement is plain: a single-cloud API is simplest and fastest when the data is allowed to leave the building, and an on-device model is simplest when the task is small enough for one machine. Federated inference earns its coordination overhead only in the gap between those two — sensitive data, large-model ambition, several parties who don't fully trust each other.
Where it breaks
Federated inference reduces, but does not eliminate, the ways sensitive data can leak — and it adds its own new failure surfaces.
Not a solved privacy guarantee
A 2025 position paper on federated inference states plainly that system-level understanding remains lacking, and that privacy-preserving inference, ensemble-based collaboration and incentive alignment across owners are recurring friction points, not settled engineering. Treat any vendor claim of "fully private" federated inference as provisional.
Latency is the other recurring cost. Splitting one model's computation across several machines means several network hops where there used to be none; if those hops cross slow or distant links, the system can end up slower than a smaller model running entirely in one place. The FIRST toolkit's answer — keeping some nodes "hot" — is a mitigation, not a cure, and it only helps the scheduling layer, not the finer-grained partitioning across edge devices. Calibrating differential-privacy noise is a similar trade-off dressed as an engineering decision: enough noise to protect the input, not so much that the final answer degrades, and the dossier gives no settled formula for getting that balance right.
Numbers to carry
| Figure | What it means |
|---|---|
| 10 September 2025 | Publication date of the ACM SIGCOMM 2025 poster introducing capability-aware model partitioning with differential-privacy noise on activations |
| "billions of tokens daily" | Reported scale of on-premises inference generated across federated HPC clusters using the FIRST toolkit, without commercial cloud infrastructure |
| 2025 | Year the first formal position-paper definition of federated inference, with its three confidentiality requirements, was published |
| 2026 | Year Lenovo, IBM and NVIDIA jointly published their Sovereign AI Inference solution brief |
Myths worth killing
| Commonly repeated | What is actually true |
|---|---|
| "Private AI just means a small model running on one device" | Federated inference often splits one large model's computation across several devices; privacy comes from the split and the noise, not from isolation alone |
| "Keeping input local is enough to call something federated inference" | The formal definition also requires intermediate activations and model parameters stay unexposed — raw-input privacy alone doesn't qualify |
| "Sovereign AI infrastructure and federated inference are the same thing" | Sovereign-boundary serving keeps one organisation's data inside its own fence; federated inference is specifically about splitting computation across several mutually distrustful parties |
| "Differential-privacy noise is a solved parameter you just set once" | A 2025 position paper still lists privacy-preserving inference and calibration as open friction points, not settled defaults |
| "Federated inference is the same as federated learning" | Federated learning trains a shared model across distributed data; federated inference serves an already-trained model at request time, and the two papers in the dossier treat them as distinct, complementary paradigms |
Worked example
- Three hospital sites each hold patient notes locally and refuse to send raw notes to an external API for a discharge-summary model.
- An orchestrator profiles each site's available compute and splits the model's layers accordingly — illustrative, the strongest site might carry 60% of the layers, the two smaller sites the rest, roughly matching the pattern of capability-aware partitioning from the SIGCOMM 2025 poster.
- Each site runs its slice of the model over its own patient notes locally; no site transmits plaintext notes to another.
- Before sending its intermediate activations onward, each site injects calibrated noise into them, following the same noised-transmission principle described for edge devices.
- A combining step, modelled on the KV-cache-sharing approach in FedRefine, merges the sites' partial computations into one discharge summary without any site seeing another's full patient notes or full model weights.
- The hospital network's IT team monitors the result for a drop in summary quality against a known baseline — illustrative, a loss of a few percentage points in clinical-accuracy spot checks would be the signal to re-tune the noise level, since the dossier gives no fixed threshold for this.
Should you use it?
Use federated inference when the answer genuinely must be computed on data that policy forbids centralising, and when no single node you control has the compute to run the full model alone. Skip it when a single trusted cloud is already permitted — the coordination overhead buys nothing there — or when a modest on-device model is simply good enough for the task.
The cheapest first experiment is not to build a full multi-party system: it's to measure, on one workload, how much quality is lost when noise is added to intermediate values at the level the SIGCOMM 2025 framework describes, before deciding whether partitioning across real organisational boundaries is worth the engineering.
Say it in the room
"Use federated inference when the data legally can't leave several separate buildings and one big model still needs to see all of it; skip it when one trusted cloud is already permitted."
Lines to carry
- "Privacy by architecture, not by isolation."
- "Share the blurred page, never the diary."
- "Each node carries only the part it can be trusted with."
- "Boundary-respecting is not the same as multi-party-private."
- "First-stage question: does any single machine see the whole thing?"
- "Noise calibration is still an open problem, not a solved default."
FAQ
Is federated inference the same as federated learning?
No. Federated learning trains a shared model using data kept on many separate devices. Federated inference happens after training — it's about running the already-trained model collaboratively at request time, with no single node seeing the full input, model or output.
Does keeping data on-device automatically make something federated inference?
Not by the formal definition. A 2025 position paper requires that input, model parameters, and intermediate activations all stay unexposed to other participants — keeping raw input local alone doesn't satisfy the other two conditions.
Is this the same as what people call "sovereign AI"?
Related but not identical. Sovereign AI is the broader goal of keeping data, models and operations under an organisation's or nation's own control. Federated inference is one architectural mechanism that can help reach that goal by splitting computation across controlled nodes, rather than relying on a single external endpoint.
Why add noise to intermediate activations instead of just encrypting them?
Encryption and noise solve slightly different problems; the SIGCOMM 2025 framework uses calibrated differential-privacy noise specifically because activations can be used to reconstruct the original input, and noise gives a measurable privacy guarantee at a known accuracy cost, rather than requiring heavier cryptographic computation at every hop.
Does federated inference slow everything down?
It can. Splitting one model's computation across several machines introduces network hops that didn't exist before. Scheduling tools like FIRST try to mitigate this by keeping frequently used nodes "hot", but the dossier doesn't claim this removes the latency cost entirely — only that it reduces it for pooled, federated clusters.
Glossary
- Federated inference — running a trained model's prediction step across multiple separate computers or organisations so no single one sees the full input, model or output.
- Federated learning — training a shared model using data kept on many devices, without centralising the raw data.
- Model partitioning — splitting a large model's layers across different devices so each handles only part of the computation.
- Differential privacy noise — calculated random noise added to data so individual details can't be reverse-engineered while overall patterns stay useful.
- Intermediate activations — the numerical values a model computes partway through processing, which can still leak information about the original input.
- KV cache (key-value cache) — stored intermediate values reused to speed up generating further output, especially in large language models.
- Sovereign AI — keeping an organisation's or nation's AI data, models and operations under its own control.
- Edge device — a computer near where data is generated, such as a phone or local server, rather than in a remote data centre.
- Orchestrator — a central component that decides how a task is divided and assigned among participating nodes.
- Inference-as-a-Service — offering access to running pre-trained models for prediction, delivered like a cloud service.
- HPC cluster — a group of powerful connected computers used for demanding computational workloads.
- Ensemble-based collaboration — combining outputs from several separate models to produce a better result than any one alone.
- Non-IID data — data that isn't identically distributed across sources, so each node's data may look quite different from another's.
Sources
- Federated Inference: Towards Collaborative and Privacy-Preserving Inference over Edge Devices — ACM SIGCOMM 2025 Posters and Demos, 2025
- FIRST: Federated Inference Resource Scheduling Toolkit for Scientific AI Model Access — arXiv, 2025
- FedRefine: Federated Refinement for Heterogeneous LLM Collaboration via KV Cache Communication — arXiv, 2025
- P2P / Federated Inference — LocalAI
- Federated Inference: Toward Privacy-Preserving Collaborative and Incentivized Model Serving — arXiv, 2025
- Own Your Own Intelligence: The Lenovo Sovereign AI Inference Solution with IBM and NVIDIA — Lenovo Press, 2026