DM Explains · No. 047 · Agents
Stateful agent federation: letting AI agents remember their work across crashes and teams
Agents are starting to keep a durable, shareable memory of what they did, so a crash or handoff no longer means starting the task over — but that durability costs real speed.
· 14 min read
Read me first
Stateful agent federation genuinely does let agents survive crashes and handoffs without starting over, but only because every step gets written to a durable journal first — and that writing, especially across a network, is never free.
A research agent gathers facts across tool calls → 1 each call recorded as a transaction with pre-state, intent, effect → 2 commit marker written only after durable persistence confirmed → 3 state shared via PD-KVC over RDMA instead of recomputed → 4 federated handoff to a helper agent costs network time, p50 RTT split across request/response paths → 5 helper replays the journal from last commit to find unfinished work → 6 PANDA-style clustering assembles a team of agents in milliseconds, sustaining 100% task completion under faults
Meera is halfway through baking a cake when the power cuts. She doesn't throw out the batter; she already wrote down exactly which ingredients went in and which step she'd reached, so when the lights come back Amma can pick up from that exact point rather than starting the whole cake over. Long-running AI agent tasks face the same cliff edge every time a network blips or a process dies mid-task, except most of them have no such note.
A multi-step agent workflow can run for minutes or hours, calling many tools and handing work to other agents along the way. If the infrastructure underneath treats every step as stateless — as most agent frameworks still do by default — then one crash discards all the reasoning and tool-call history built up so far, and the task restarts from zero, often silently. The fix is to write everything down before it happens and confirm it stuck after, but passing that record across machines and organisations is measurably slower than keeping it in one process.
Everyday picture. Meera's cake-making stops when the power cuts. Because she noted exactly which ingredients went in and which step she'd reached, Amma can resume baking without redoing any of it. Dadi keeps a shared kitchen notebook so whoever cooks next — Papa, Arjun, or whoever feeds Bruno — knows what's already done.
Where the picture breaks. A recipe has one fixed sequence of steps; real agent workflows branch into many possible next actions, and the journal has to capture which branch was taken, not just a step number. The notebook also stays in one kitchen; federated agent memory has to stay consistent across many separate kitchens — different machines, different organisations — joined only by a network.
Why it exists now
Most agentic AI frameworks are stateless by design, or leave state management entirely to whoever builds on top of them — convenient for a single question-and-answer exchange, a liability for a task that unfolds over many turns. Agent interactions are inherently stateful in a different sense than a web request: they need short-term retention of context within one interaction and long-term persistence across many, which is a harder problem than most existing agent protocols were built to solve (Position: Collaborative Agentic AI Needs Interoperability Across Ecosystems, arXiv, 2025).
What changed is that infrastructure groups started treating this as a fabric-level problem rather than an application-level patch. In October 2026, the Open Compute Project's AI Fabric workgroup published the "AI Fabric for Agentic AI Whitepaper 2026-v1.0", proposing concrete primitives — Agent State Journaling and a Persistent Distributed KV Cache — specifically for long-running, stateful agent workloads, extending an earlier framework that had only covered stateless inference and training (AI Fabric for Agentic AI Whitepaper 2026-v1.0, Open Compute Project, 2026). Around the same time, academic groups published decentralised multi-agent architectures tested at scale, federated middleware for scientific computing, and position papers arguing that agent-to-agent protocols need standardised state management the way databases have had write-ahead logs for decades. Taken together, this marks a shift from research demos to engineering durability directly into the plumbing agents run on.
How it works
Record as transaction
Write down the batter before you taste the cake
What it is. Each unit of agent progress gets wrapped into an Agent Transaction: a before-snapshot of context, memory references and tool history; a record of what the agent intended to do; a record of what actually happened; and a commit marker.
Why it exists. This mirrors write-ahead logging in distributed databases, so the agent's causal history — not the raw conversation transcript — becomes the thing recovery trusts.
Where it breaks. If the snapshot or intent record is incomplete, recovery can't distinguish what the agent meant to do from what actually occurred.
How that's handled. The Agent State Journaling (ASJ) model defines exactly four parts to this transaction, so nothing gets recorded halfway.
In technical terms
In technical terms: the Agent State Journaling model composes an Agent Transaction from a pre-state snapshot, an intent record, an effect record, and a commit marker, modelled on write-ahead logging (AI Fabric for Agentic AI Whitepaper 2026-v1.0, Open Compute Project, 2026).
Confirm it stuck
Don't call it done until it's durable
What it is. The commit marker in that transaction is only written once the effect record is confirmed to have been durably persisted by the underlying fabric.
Why it exists. Without this guarantee, a crash between acting and recording leaves the agent's state ambiguous — did the tool call actually fire, or not?
Where it breaks. Confirming durability itself takes time, and it ties the agent's speed to whatever storage or network layer sits underneath it.
How that's handled. Durability is bought, not given free: the system accepts the extra wait in exchange for a state it can trust after a failure.
In technical terms
In technical terms: the commit marker is written strictly after the fabric acknowledges durable persistence of the effect record, closing the gap between an action occurring and its being recoverable (AI Fabric for Agentic AI Whitepaper 2026-v1.0, Open Compute Project, 2026).
Share the memory
A shared notebook beats rebuilding the recipe
What it is. A proposed vendor-neutral Persistent Distributed KV Cache (PD-KVC) lets an agent's working memory — its key-value cache — be migrated or shared across machines rather than staying locked inside one process.
Why it exists. Recomputing a long agent context from scratch is expensive; a shared, persistent memory tier means another agent, or a recovery process after a crash, can resume from the same point instead of starting over.
Where it breaks. Keeping shared memory consistent across many machines is a much harder problem than one notebook sitting in one kitchen.
How that's handled. The cache is integrated directly into the datacentre fabric using RDMA-class transports, rather than bolted on as an application-level cache.
In technical terms
In technical terms: PD-KVC is proposed as integrated with the AI fabric via RDMA-class transports including UEC 1.0 and UALink, so KV-cache state can move between hosts without full recomputation (AI Fabric for Agentic AI Whitepaper 2026-v1.0, Open Compute Project, 2026).
Cross the network
Lending notes to another school costs time
What it is. When state has to move to an independently-run agent — a different process, machine, or organisation — it travels over a federated messaging path rather than staying inside one local system.
Why it exists. Federation is what lets agents owned by separate teams or organisations cooperate on the same task at all.
Where it breaks. A study of persistent agent interaction over federated messaging found the federated path running at only 76.5-78.4% of the corresponding local-only performance, with the added latency split almost evenly between request-path and response-path time, and executor time contributing less than 0.1% of p50 round-trip time — the cost sits almost entirely in the network, not the computation (Persistent Agent Interaction over Federated Messaging, ResearchGate, 2026).
How that's handled. Federation adds a network between every handshake, so systems that need it accept the tax deliberately rather than discovering it by accident in production.
In technical terms
In technical terms: federated agent messaging introduces request- and response-path latency that dominates round-trip time, measured at 76.5-78.4% of local-only performance in one 2026 study (Persistent Agent Interaction over Federated Messaging, ResearchGate, 2026).
Illustrative: research agent crashes after 3 of 5 tool calls → 1 last commit found at call 2 → 2 call 3 marked in-flight, calls 4-5 not started → 3 replay reconciles call 2's effect from the journal → 4 authority rebinds to a helper agent → 5 handoff over the federation network → 6 helper resumes from call 3 → task completed with all 5 calls accounted for
Detect and replan
Read the notebook to know what's left to cook
What it is. When a component or agent fails, the system reads the journal back from the last committed transaction to work out which actions finished, which were mid-flight, and which never started.
Why it exists. Coarse, application-level checkpointing can't make these distinctions; it only knows "something was saved a while ago", not "this exact tool call completed and that one didn't".
Where it breaks. Without fine-grained, causally-aware records, replanning risks silently redoing an action that already had a side effect — sending the same email twice, charging a card twice.
How that's handled. Fine-grained replay combined with decentralised orchestrators that dynamically reassemble teams around a failure let systems like PANDA sustain 100% task completion under injected faults where existing systems failed (PANDA: A Decentralized Architecture with Flexible Orchestration for Scalable, Fault-Tolerant Multi-Agent Systems, arXiv, 2026). Separately, a runtime-independent persistent agent design proposes six continuity invariants and a quiesce-checkpoint-validate-bind-rehydrate-resume protocol to govern exactly this kind of recovery (Runtime-Independent Persistent Agents, arXiv, 2026).
In technical terms
In technical terms: recovery proceeds by replaying the Agent Transaction journal from the last commit marker, classifying each prior action as completed, in-flight, or not-started, before any replanning begins (AI Fabric for Agentic AI Whitepaper 2026-v1.0, Open Compute Project, 2026).
Form a team
Thousands of cooks, small kitchens
What it is. Large pools of agents discover each other's capabilities and cluster into small, task-specific teams rather than routing everything through one central coordinator.
Why it exists. A single orchestrator becomes a bottleneck and a single point of failure once the number of agents grows; self-organisation decouples collective-wide coordination from team-level coordination.
Where it breaks. Self-organisation depends on reliable capability discovery; a poorly described agent can get matched into the wrong team entirely.
How that's handled. PANDA reports scaling to thousands of agents, assembling teams in milliseconds, matching state-of-the-art accuracy at up to 8x the efficiency of existing systems (PANDA, arXiv, 2026). The Federation of Agents framework uses smart clustering that groups agents working on similar subtasks for the same reason (Federation of Agents: A Semantics-Aware Communication Fabric for Large-Scale Agentic AI, arXiv, 2025).
In technical terms
In technical terms: PANDA decouples collective-wide communication from team-level communication, letting team assembly happen in milliseconds at thousands-of-agents scale while matching accuracy at up to 8x the efficiency of existing systems (PANDA, arXiv, 2026).
Compared with what came before
| Stateless orchestration | Stateful agent federation | |
|---|---|---|
| What survives a crash | Nothing — the task restarts from the beginning | Everything up to the last committed transaction |
| Where memory lives | Inside one running process | Shared and persisted, can migrate across machines |
| Failure handling | Application-level retry logic, if any | Journal replay distinguishes completed, in-flight and not-started actions |
| Speed under normal conditions | Fastest, no coordination overhead | Measurably slower — 76.5-78.4% of local-only performance in one 2026 study of federated messaging |
| Scaling to many agents | Central orchestrator becomes a bottleneck | Self-organising teams assembled in milliseconds at thousands-of-agents scale, per PANDA |
Stateless orchestration is simpler and faster precisely because it promises nothing after a failure. Stateful federation trades some of that speed for a guarantee: the work already done is not thrown away. Whether that trade is worth it depends entirely on how expensive redoing the work would be.
Where it breaks
If a task is short, single-turn, and cheap to redo, none of this machinery earns its keep — the journaling, confirmation and network hops only add latency to something that didn't need protecting.
Not a security boundary
Durable journaling and shared memory govern what an agent remembers and can resume, not who is allowed to see it. Access control still has to be enforced separately, before state is shared or migrated to another agent or organisation.
Latency-critical interactions are a poor fit: even the modest-sounding drop to 76.5-78.4% of local-only performance, measured in one 2026 study of federated agent messaging, can be unacceptable when a user is waiting on the other end (Persistent Agent Interaction over Federated Messaging, ResearchGate, 2026). And self-organising team formation depends on agents describing their own capabilities accurately; a vague or wrong description can place an agent into the wrong team, a failure mode that no amount of journaling fixes, because the problem happened before any transaction was recorded.
Numbers to carry
| Figure | Detail |
|---|---|
| 76.5-78.4% | Federated-path performance relative to local-only, in a 2026 study of persistent agent interaction over federated messaging |
| Less than 0.1% of p50 RTT | Share of round-trip time attributable to executor computation in that same study — the rest is network |
| Thousands of agents | Scale PANDA is reported to coordinate, assembling teams in milliseconds |
| Up to 8x | Efficiency gain PANDA reports over existing systems at matching accuracy |
| 100% task completion | PANDA's reported completion rate under injected faults, where existing systems failed |
| 4 components | Parts of an Agent Transaction in the Agent State Journaling model: snapshot, intent, effect, commit marker |
| 6 continuity invariants | Number defined by a proposed runtime-independent persistent agent protocol governing state preservation |
| October 2026 | Publication of the "AI Fabric for Agentic AI Whitepaper 2026-v1.0" |
Myths worth killing
| Commonly repeated | What is actually true |
|---|---|
| "A long context window is the same as persistent memory" | A context window only holds state within one running process; federation persists and migrates that state across processes, machines and organisations, surviving crashes |
| "Checkpointing already solves this" | Generic checkpointing saves coarse snapshots; Agent State Journaling is fine-grained and causally aware, recording intent and effect separately so recovery knows exactly what completed |
| "Federated agents are basically federated learning" | Federated learning shares model gradients to train a model; stateful agent federation shares live task context and progress between running agents, not training signals |
| "More agents means slower coordination" | PANDA reports assembling teams of thousands of agents in milliseconds by decoupling collective-wide from team-level communication |
| "Durability is free once you've built the journal" | The journal only helps once persistence is confirmed; that confirmation step, and any network hop to share state, adds measurable latency every time |
Worked example
- A research agent (the running task) starts gathering facts across several tool calls, each wrapped into an Agent Transaction with a before-snapshot, an intent record and an effect record.
- After each tool call completes, the fabric confirms durable persistence and only then writes the commit marker — illustrative: say step three of five has just been committed when the process crashes.
- A recovery process reads the journal back from the last commit marker, sees that steps one through three completed, step four was mid-flight with no effect record, and step five never started.
- Recovery replays from step four rather than restarting the whole task, because the journal distinguished "in-flight" from "done".
- Partway through, the task hands off to a second, independently-run helper agent; that handoff travels over a federated messaging path, arriving at roughly 76.5-78.4% of the speed it would have taken had both agents shared one local process.
- The helper agent reads from the shared, persistent memory tier rather than recomputing the research agent's earlier context from scratch, and continues the task with its progress intact.
Should you use it?
Use stateful agent federation when a workflow is long-running, multi-turn, and spans multiple tools, sub-agents or organisations, where a failure mid-task must not discard already-completed reasoning. Skip it for short, single-turn requests with nothing to resume, and for latency-critical interactions where even a drop to roughly three-quarters of local-only speed is unacceptable. The cheapest first experiment is to add transaction-style logging — snapshot, intent, effect, commit — to one existing multi-step agent task, before touching federation at all, and measure how often a crash actually loses work worth protecting.
Say it in the room
"Use it when the answer lives in a task that takes minutes or hours and must survive a crash or handoff; skip it when the task is short enough to simply retry."
Lines to carry
- "Durability is bought, not given free."
- "A journal remembers what happened, not just what was said."
- "Shared memory beats recomputed memory."
- "Federation adds a network between every handshake."
- "Thousands of agents, small teams, fast assembly."
- "Measure what a crash actually costs before you pay to prevent it."
FAQ
Isn't this just checkpointing with a new name?
Not quite. Ordinary checkpointing saves a coarse snapshot of an application every so often; Agent State Journaling records each step as a transaction with a separate intent and effect, so recovery can tell exactly which action completed versus which merely started. That distinction matters once an action has a side effect in the world, like sending a message.
Does federation always make things slower?
In the one study measured here, yes — the federated path ran at 76.5-78.4% of local-only performance, with the extra time sitting almost entirely in the network rather than in computation. That's a real tax, not a rounding error, and it's worth deciding up front whether the durability is worth paying it.
Can a single chatbot session use any of this?
Not meaningfully. A single long context window already keeps state within one running process; the problems journaling and federation solve only appear once state has to survive a crash of that process or move to a different one.
How does an agent find other agents to team up with?
By advertising capabilities that other agents or a lightweight discovery layer can match against; systems like PANDA report assembling teams from thousands of candidate agents in milliseconds. The catch is that the matching is only as good as the capability description each agent provides.
Is this the same as federated learning?
No. Federated learning shares model gradients or parameters across sites to train a model. Stateful agent federation shares live task context, memory and progress between agents that are already running, with no training involved.
Glossary
- Agentic AI — AI systems that plan, reason, and take multi-step actions, such as calling tools or other agents, over extended tasks.
- Stateful — Keeping track of information between interactions, rather than treating each one as independent.
- Federation — Connecting independently owned or operated systems so they can cooperate while remaining separately controlled.
- KV-cache — Stored intermediate computations from a language model that avoid recomputing earlier parts of a task.
- RDMA — Remote Direct Memory Access: a networking technique letting one computer read or write another's memory directly, for low latency.
- Checkpointing — Periodically saving a system's state so work can resume from that point after a failure.
- Durable execution — A pattern where a workflow's progress is persistently recorded so it can survive crashes and resume where it left off.
- Write-ahead log — A record of intended changes written before they are applied, used for crash recovery.
- Agent Transaction (AT) — The atomic unit of agent progress in the ASJ model: pre-state snapshot, intent record, effect record, commit marker.
- Journal replay — Reconstructing state after failure by reading recorded history and verifying each step.
- PD-KVC (Persistent Distributed KV Cache) — A proposed shared, persistent store for KV-cache memory accessible across machines.
- ASJ (Agent State Journaling) — A proposed model for recording an agent's actions and outcomes in a fine-grained, causally-aware log.
- RTT (round-trip time) — The total time for a request to travel to a destination and its response to return.
- HotPotQA — A multi-step question-answering benchmark used to test multi-agent systems.
- UEC / UALink — Emerging industry networking standards for high-speed, low-latency interconnects in AI datacentres.
- A2A — Agent-to-agent communication: protocols letting separate AI agents exchange messages or tasks.
Sources
- AI Fabric for Agentic AI Whitepaper 2026-v1.0 — Open Compute Project, 2026
- Persistent Agent Interaction over Federated Messaging: Recovery and Application-Path Performance — ResearchGate, 2026
- Federation of Agents: A Semantics-Aware Communication Fabric for Large-Scale Agentic AI — arXiv, 2025
- Position: Collaborative Agentic AI Needs Interoperability Across Ecosystems — arXiv, 2025
- Runtime-Independent Persistent Agents — arXiv, 2026
- PANDA: A Decentralized Architecture with Flexible Orchestration for Scalable, Fault-Tolerant Multi-Agent Systems — arXiv, 2026
- Empowering Scientific Workflows with Federated Agents — IEEE IPDPS, 2026