DM Explains · No. 042 · Orchestration
Durable workflows: how AI agents survive crashes without repeating themselves
Agents crash mid-task all the time — the question is whether they double-send the email when they come back. Durable workflows are how production systems answer that.
· 14 min read
Read me first
Agents crash mid-task constantly — a serverless function gets killed, a network blips, a process times out. That part is unavoidable. What's avoidable is what happens next: whether the agent re-sends the email, re-charges the card, or rewrites the page it already finished. Durable workflows genuinely fix that, but only when every step writes down what it meant to do before it does it.
An agent opens a pull request and waits on CI → 1 the CI-wait call is isolated from the rest of the task → 2 an intent record is written before the wait begins → 3 an effect record is committed only once CI reports pass/fail (f8) → 4 process is killed; another resumes from the last completed step (f5) → 5 the wait is marked replay-safe, so it resumes automatically rather than re-triggering CI (f7) → 6 the orchestration view shows the task still waiting on CI, not restarted from PR open
A coding agent opens a pull request, writes a pricing page, and sits waiting on continuous integration to finish. Nothing unusual there, except that the cloud function running it gets killed mid-wait — which is simply what serverless functions do when they time out or get rescheduled. The pricing page is already written. The pull request is already open. Only the waiting is left.
Without any plumbing for this, the agent just stops. Someone restarts it later, and the restart has no memory of what already happened — so it rewrites the pricing page, and depending on how the tool calls are wired, it may re-trigger side effects like a second email or a duplicate charge. In a demo that's an eye-roll. In production, that's a customer billed twice for the same thing.
Everyday picture. Dadi is halfway through writing a long letter when the power cuts out. Because she keeps a notebook recording each sentence as she finishes it, she picks up exactly where she left off instead of starting the letter over.
Where the picture breaks. Dadi's notebook only tracks her own writing — it cannot un-send a letter she already posted. Durable workflows have to handle exactly that case: external side effects, like emails already sent, that no amount of restarting can undo.
Why it exists now
Agents don't just generate text anymore; they call tools, write to databases, send emails and trigger payments across tasks that run for minutes, hours or days. A plain retry-on-failure — catch the error, run the whole thing again — assumes every step is safe to repeat. That assumption held reasonably well for stateless API calls. It does not hold for an agent that has already sent a parcel.
Through 2025 and 2026 several platforms shipped durability specifically for this shape of problem. Cloudflare's Workflows documentation describes a pattern for building a durable AI agent, where each large language model call and each tool call becomes a self-contained, individually retryable step (Cloudflare). Mistral released Mistral Workflows, built on Temporal, an open-source engine for fault-tolerant workflow orchestration, for production AI workflows (Mistral Docs). Vercel published an open-source package called durable-harness in 2026, aimed at making any coding-agent harness — Claude Code, Codex, OpenCode — resumable after a crash (Vercel). And the Open Compute Project's 2026 whitepaper on AI infrastructure proposed standardising this at the fabric level, under the name Agent State Journaling (Open Compute Project).
That convergence across several unrelated vendors, on roughly the same shape of fix, is the signal worth noticing. None of it is new computer science. Write-ahead logging and event sourcing have existed in databases for decades. What changed is that agents now do enough real-world things, for long enough, that borrowing those old reliability patterns stopped being optional.
How it works
Step isolation
Each call stands alone, so one failure doesn't sink the task
What it is. Every model call or tool call is treated as its own retryable unit of work, rather than one line inside a long unbroken function.
Why it exists. If the whole task is one function, one failure anywhere means redoing everything. If each call is its own step, a failure only affects that step.
Where it breaks. Isolation tells you a step can be retried; it doesn't tell you whether an interrupted step already happened out in the world. That needs a record, not just a boundary.
How that's handled. The next layer writes intent before the step runs.
In technical terms
In technical terms: Cloudflare's Workflows documentation describes each LLM call and tool call as a self-contained, individually retryable step, with automatic retries on step failure or resumption from the last successful step after a crash (Cloudflare).
Intent logging
Write down what you're about to do before you do it
What it is. Before a call runs, the system writes an entry describing what it is about to attempt — which tool, which parameters — into a durable log.
Why it exists. If the process dies mid-call, whoever recovers it needs to know what was being attempted, not just that something was running.
Where it breaks. An intent record proves something was about to happen. It does not prove it's safe to simply run again — a sent parcel can't be unsent just because the log says "sending parcel" was attempted.
How that's handled. Intent is paired with a separate record of whether the effect actually landed, which is the next step.
In technical terms
In technical terms: Temporal states its workflows automatically capture state at every step and, on failure, can resume exactly where they left off (Temporal). The Open Compute Project's proposed Agent Transaction structure records a pre-state snapshot and an intent record before execution (Open Compute Project).
Effect and commit
Nothing counts as done until it's durably confirmed done
What it is. Once a call finishes, an effect record captures its output, and only after that record is durably persisted does a commit marker get written.
Why it exists. This separates "attempted" from "confirmed complete". A step isn't done just because the model returned text; it's done once that result is safely on disk somewhere a crash can't erase.
Where it breaks. This protects a step that fully completed — it won't get billed or executed twice. It says nothing about a step that was still mid-flight when the crash hit; that's a different problem, handled by classification rather than commit records.
How that's handled. Separate replay-safety rules, covered in step five, decide what to do with calls that were interrupted rather than completed.
In technical terms
In technical terms: Vercel's durable-harness writes an intent record before a model or tool call runs and an outcome record when it finishes, so a completed model call's tokens are never billed twice (Vercel).
Resumption
Pick up the task, not restart it
What it is. When a process dies, a new process reads the log and continues from the last completed step, instead of redoing the whole task from the beginning.
Why it exists. This is the entire point of the exercise: the coding agent's pricing page was already written and shouldn't be written again just because the process restarted mid-wait on continuous integration.
Where it breaks. Resumption only works if the log itself survived the crash. A crash that happens mid-write to the log — not mid-step, but mid-record — is the edge case every real implementation has to account for separately.
How that's handled. Logs are written to durable storage before a step is allowed to proceed, so the log's own write is treated as the thing that must succeed first.
In technical terms
In technical terms: Mistral Workflows, built on Temporal, records every step in an event history so another process resumes from the last completed step if one dies mid-workflow (Mistral Docs). Vercel demonstrated this directly with a coding agent resuming its wait on continuous integration after the function running it was killed (Vercel).
Illustrative: the coding agent waiting on CI is killed mid-wait → 1 intent "waiting on CI run" was already recorded → 2 effect uncertain: did the wait callback fire? → 3 classified unsafe (polling CI again could double-trigger a webhook) → 4 (skipped, not safe to retry blindly) → 5 escalated: the model is told the wait may have resolved and checks CI status itself → resumed exactly at the wait, not at step one of the pull request
Replay safety
Not every action is safe to just try again
What it is. When a step is interrupted — not cleanly finished, not cleanly failed, just cut off — the system has to decide whether it's safe to run again.
Why it exists. Some actions are idempotent: running them twice produces the same result as running them once, so an automatic retry is harmless. Others aren't — sending a message, charging a card. Treating those the same way duplicates real-world side effects.
Where it breaks. Classification depends on someone correctly marking which actions are idempotent in the first place. A wrongly labelled "safe" action can still duplicate a side effect, silently.
How that's handled. Anything not explicitly marked safe gets reported back to the model as possibly already run, rather than blindly retried, so the model (or a human) can check before repeating it.
In technical terms
In technical terms: in Vercel's durable-harness, calls marked replay: 'safe' are automatically rerun, while any other interrupted call is reported to the model as possibly already run, leaving the decision to the model rather than the retry mechanism (Vercel).
Visible orchestration
Keep the map separate from the territory
What it is. The workflow's overall structure — which steps exist, in what order, with what retries — is kept as an inspectable, versioned graph, separate from the workers actually executing side effects.
Why it exists. Separating the map from the territory lets an operator pause, resume, retry or terminate a specific run according to policy, and lets anyone audit what a given run actually did.
Where it breaks. This visibility is infrastructure, and infrastructure has a cost and a learning curve that a one-off script simply doesn't need.
How that's handled. Treat it as a deliberate trade: adopted once the task has enough steps and enough external consequence to justify it, not before.
In technical terms
In technical terms: Conductor persists every workflow step with configurable retries and timeouts (Conductor OSS), and the Open Compute Project's whitepaper cites LangGraph's checkpointer, Microsoft Agent Framework's Checkpoint Manager, OpenAI Agents SDK's durable-execution integration, Pydantic AI's Temporal binding and the Dapr Agents framework as convergent industry implementations of the same idea (Open Compute Project).
Compared with what came before
| Plain retry logic | Durable workflow | |
|---|---|---|
| Unit of recovery | The whole function or request | A single logged step |
| Knows what already completed | No — reruns from the top | Yes — reads the log |
| Safe with non-idempotent actions | Only if the caller is careful | Classified explicitly, unsafe ones escalated |
| Survives a process crash, not just an error | Not reliably | Designed for exactly this |
| Infrastructure required | Minimal | A durable log and an orchestration layer |
The judgement is not that plain retries are wrong — they're the correct choice for a single stateless call with no external side effect. The judgement is that they quietly stop being correct the moment a task has several dependent steps and at least one of them sends something, charges something, or writes something that can't be casually overwritten.
Where it breaks
Not a security boundary
Durable workflows decide what gets retried and in what order; they don't decide who is allowed to trigger the workflow in the first place or what it's allowed to touch. Permissions still have to be enforced separately, before a step runs, not inferred from the fact that it's logged.
The ACM Queue analysis of fault-tolerant AI agents argues that many agent failures — timeouts, partial writes, duplicated side effects, cascading retries — get misdiagnosed as model failures, when they're actually distributed-systems problems with distributed-systems fixes (ACM Queue). That's a reasonable diagnosis, but it cuts both ways: adding durability machinery to a task that never had these failure modes just adds operational weight for no benefit.
The Open Compute Project's whitepaper also flags a subtler gap: current application-level checkpointing is coarse-grained, typically taken at step boundaries, so a failure partway through a step still loses that step's partial work (Open Compute Project). Durability reduces how much work is lost on a crash; it doesn't reduce it to nothing unless steps themselves are kept small.
And the whitepaper's own motivating evidence is blunt about why any of this matters: early agentic deployments, including enterprise automation and autonomous coding assistants, have reported high failure rates on complex multi-step tasks from model errors, tool call timeouts, network partitions and hardware failures — forcing operators into manual, application-level patches rather than something built into the platform (Open Compute Project).
Numbers to carry
| Item | Detail |
|---|---|
| Vercel's durable-harness release | Open-sourced in 2026, covering Claude Code, Codex and OpenCode harnesses |
| Mistral Workflows foundation | Built on Temporal, an open-source fault-tolerant workflow engine |
| Cloudflare Workflows retry unit | Each LLM call and tool call is its own individually retryable step |
| Replay classification in durable-harness | Calls marked replay: 'safe' auto-rerun; all others are reported to the model as possibly already run |
| Open Compute Project checkpoint granularity finding | Checkpointing is typically taken at step boundaries, so mid-step work is lost on failure |
Myths worth killing
| Commonly repeated | What is actually true |
|---|---|
| "A retry loop is basically the same thing as a durable workflow" | A retry loop reruns a whole function with no record of which sub-steps already completed; a durable workflow logs each step and resumes only the unfinished part |
| "If the model is smarter, it won't double-send things" | Double-sends happen because the process restarts with no memory of the log, not because the model made a reasoning error |
| "Checkpointing in training solves this too" | Training checkpoints restart pure computation with no external consequence; agent steps can send emails or charge cards, which training-style checkpointing never has to account for |
| "Everything can just be retried safely" | Only idempotent actions can; durable-harness explicitly separates calls marked safe to replay from everything else, which gets escalated instead of rerun |
Worked example
- The agent opens a pull request and writes a pricing page, then calls the tool that triggers continuous integration and starts waiting.
- The durability layer writes an intent record before the wait begins, noting that the agent is waiting on a specific CI run for a specific pull request.
- The cloud platform kills the function mid-wait, as serverless platforms routinely do on timeout or rescheduling.
- A new process starts, reads the durable log, and sees the last completed step was "pull request opened, pricing page written" and the in-progress step was "waiting on CI run X".
- The replay-safety check classifies "wait on CI run X" as safe to resume rather than something that needs re-doing, because waiting doesn't duplicate any side effect.
- The agent resumes watching that same CI run instead of rewriting the pricing page or reopening a second pull request — illustrative detail: in a version without any of this machinery, the naive restart would simply rewrite the page from scratch and could open a second pull request.
Should you use it?
Use durability once a task runs long enough, and touches enough real external systems, that a crash halfway through would cost more than the plumbing to prevent it. A task that sends money, sends messages, or writes to a system of record is the clearest case. A single short-lived call with no external side effect is not — plain retry-on-failure is enough, and adding a durable log to it is pure overhead.
The cheapest first experiment is not to adopt a whole orchestration platform. It's to take one existing multi-step agent task, pick the one step in it that has a real external side effect, and make that one step idempotent and logged before touching anything else.
Say it in the room
"Use it when a crash mid-task would mean a duplicated email or a double charge; skip it when the worst a crash can do is make you press run again."
Lines to carry
- "Write down what you meant to do before you do it."
- "Nothing is done until it's durably confirmed done."
- "Resume the task, don't restart it."
- "Not every action is safe to try twice."
- "The fix for flaky agents is plumbing, not prompting."
FAQ
Is this just retry logic with a different name?
No. Retry logic reruns a whole function with no memory of which sub-steps already finished. Durable workflows log each step separately, so a resumed run only redoes the part that didn't complete — and that distinction is exactly what stops a finished pricing page from being rewritten.
Does this require Temporal specifically?
No — Temporal is one well-established engine behind this pattern, used by Mistral Workflows, but Cloudflare Workflows, Conductor, and several checkpoint systems cited in the Open Compute Project's whitepaper implement similar ideas on their own infrastructure.
What happens to a tool call that was mid-flight when the crash hit?
It's classified: calls marked safe to replay get rerun automatically, and anything else gets reported back to the model as possibly already run, so the model or a human can check before repeating it, rather than the system guessing.
Does durability make an agent correct?
No. It makes an agent's progress survive a crash without duplicating side effects. Whether the agent's decisions were good in the first place is a completely separate question that durability doesn't touch.
Is this overkill for a small prototype?
Usually, yes. The dossier's own framing is that this fits tasks running minutes to days with real external side effects; a prototype with no external consequences is better served by simple retry-on-failure.
Glossary
- Durable execution — running software so its state and progress are persisted at each step, letting it survive crashes and resume rather than starting over.
- Idempotent — describes an action that produces the same result even if performed more than once, making it safe to retry.
- Checkpoint — a saved snapshot of a process's state at a point in time, used to resume from later rather than starting from scratch.
- Write-ahead log — a log that records an intended change before it is made, used to recover correctly after a crash.
- Event sourcing — storing a system's state as a sequence of recorded events rather than just its current snapshot, so history can be replayed.
- Orchestration — coordinating multiple steps, services or agents into one coherent multi-step process.
- Agent harness — the surrounding software that runs an AI agent's loop of reasoning, tool calls and responses, such as Claude Code or Codex.
- Tool call — an action where an AI model invokes an external function, API or service rather than just generating text.
- Retry with backoff — automatically trying a failed operation again, usually waiting longer between each attempt.
- Replay — re-running or reconstructing a process's history from its recorded log.
- Fan-out — splitting one task into multiple parallel sub-tasks or sub-workflows.
- MCP (Model Context Protocol) — a protocol that lets AI models call external tools and services in a standardised way.
Sources
- Temporal: Durable Execution Solutions — Temporal Technologies
- conductor-oss/conductor — Conductor OSS (GitHub)
- Build a Durable AI Agent · Cloudflare Workflows docs — Cloudflare
- Overview | Mistral Docs — Mistral AI
- Making every harness durable — Vercel, 2026
- Announcing Durable Workflow for Agents — Diagrid
- Failure Is a Feature: Engineering Fault-Tolerant AI Agents — ACM Queue
- AI Fabric for Agentic AI Whitepaper 2026-v1.0 — Open Compute Project, 2026