DM Explains · No. 043 · Orchestration
Durable agent workflows: why your AI agent should survive a crash
Agents now run for hours and call paid APIs along the way. Durable execution saves their progress outside the process, so a crash or a three-day wait for approval doesn't mean starting over.
· 12 min read
Read me first
Durable execution genuinely does stop a crash from wiping out an agent's progress, but it never stops a step from running twice — that second problem belongs to idempotency keys, not to the orchestration framework.
A refund agent qualifies a request, calls the payments API, then checkpoints state to Postgres after each super-step; the process dies mid-call, resumes via thread_id from the last checkpoint instead of step 1, pauses on interrupt() for a large refund, a human approves via Slack, and the payments call reruns with an idempotency key so it doesn't double-refund, then logs the result.
An agent is three steps from finishing a refund: it has researched the order, called the payments API, and is about to log the result when the server gets redeployed and the process dies mid-step. Without durability, the whole run restarts from the top, which at best wastes hours of paid API calls and at worst means the payments API gets called a second time and the customer is refunded twice.
Everyday picture. Dadi is halfway through knitting a scarf for the evening. Before putting it down for the night, she writes down her row count on a slip of paper. If the lights go out, she doesn't unravel the scarf — she picks up knitting from that exact row the next morning.
Where the picture breaks. Dadi writes her row count down only after finishing a row cleanly. But if an agent's equivalent note were written only after it sends a message or issues a refund, rather than before the action starts, that action could still happen twice when the lights come back on.
Why it exists now
Agents used to be a single prompt and a single reply — nothing to lose if the process died. That stopped being true once agents started calling paid APIs, waiting on human sign-off, and running for hours at a stretch. A workflow that qualifies a refund, calls a payments API, waits for an approver, and logs the result has real state worth protecting, and a redeploy can land in the middle of any of those steps.
Between July and August 2025, LangGraph formalised how it handles this. Pull request #5771, merged on 2025-08-01, added a durability mode for invoke and ainvoke. Pull request #5963, merged on 2025-08-27, added the same flag to the Python software development kit. That gave builders an explicit dial between speed and safety, on top of checkpointing and the interrupt() primitive that already existed. Since then, durable execution has become a named pattern discussed across LangGraph, Mastra, Temporal, Inngest and Pydantic AI/DBOS, with reference implementations such as ForgeFlow wiring LangGraph's StateGraph to PostgreSQL checkpointing and Slack-based approval.
How it works
Step boundaries
Only the step in flight needs redoing
What it is. The workflow is split into discrete steps or nodes rather than one long function.
Why it exists. Only the step that was running when the process died needs to be redone — everything before it is already saved, which is what makes partial progress recoverable rather than all-or-nothing.
Where it breaks. If one step itself does several external things at once, "redo the step" can still mean redoing more than intended.
How that's handled. Keep steps narrow: one external call per node where possible, so the unit of retry stays small.
In technical terms
In technical terms: LangGraph calls a single tick, where every scheduled node runs, potentially in parallel, a super-step. Breaking a workflow into super-steps means everything before the crash is already finished and saved.
Checkpointing
Progress lives outside the process, not inside it
What it is. After each step, a snapshot of the workflow's state is written to a database, keyed by an identifier for that run.
Why it exists. If state lived only in process memory, a crash or container kill during a deploy would erase all progress. Persisting it externally lets any worker, even one running newer code after a deploy, pick the run back up.
Where it breaks. The checkpoint only exists if it was actually written before the crash — which depends entirely on the durability mode chosen, covered next.
How that's handled. Use a durable backend (Postgres, not in-memory) for anything that matters, and resume by the same run identifier.
In technical terms
In technical terms: a checkpointer saves the graph state to in-memory, SQLite, or Postgres at each super-step, keyed by a thread_id. Resuming with the same thread_id reloads the last checkpoint. ForgeFlow checkpoints every node to PostgreSQL so any worker can resume any run.
Durability mode
Safety and speed trade off, and someone has to choose
What it is. An explicit setting that decides when the checkpoint write happens relative to the next step starting.
Why it exists. Not every workflow needs the same guarantee, and synchronous persistence costs latency.
Where it breaks. Picking "exit" for speed on a workflow that actually needs crash recovery quietly removes the safety net the rest of this list depends on.
How that's handled. Match the mode to the stakes: synchronous for anything touching money or irreversible actions, async or exit only where a lost run is cheap to redo.
In technical terms
In technical terms: durability="sync" persists state before the next step starts — the slowest but safest option, as described in a 2026 write-up on durable execution for AI agents. Async persists with a small window of risk. durability="exit" does not recover from mid-execution crashes at all.
Retry the unit
Resume from step 23, not step 1
What it is. On resume, only the failed or interrupted step is re-attempted, not the whole run.
Why it exists. Re-running a long agent loop from the start after a late failure would waste cost and time and could repeat already-successful side effects.
Where it breaks. Retrying a step still means that step runs again in full, including any external call it makes — which is exactly where the next two parts become essential.
How that's handled. Earlier steps stay checkpointed so the framework re-enters at the last incomplete step rather than replaying everything.
In technical terms
In technical terms: because earlier steps are already checkpointed, the framework re-enters the graph at the last incomplete step rather than replaying from the start — unlike journal-and-replay systems, which replay the whole handler and return saved results for finished steps.
interrupt() for approval
Park the run instead of holding a process open
What it is. A way to pause a workflow for a human response that might arrive hours or days later, without tying up a running process the whole time.
Why it exists. Holding a thread or server open for three days of waiting is wasteful and fragile; parking the run in storage is not.
Where it breaks. If that same approval re-invokes the graph twice — a double-click, a retried webhook — the step after the interrupt can run twice unless it's made idempotent too.
How that's handled. The run resumes only when re-invoked with the human's reply attached, and that reply path should itself be deduplicated.
In technical terms
In technical terms: calling interrupt() inside a node raises a GraphInterrupt, saves state, and surfaces a value to the caller, resuming when re-invoked with a Command carrying the human's response. ForgeFlow wires this to interrupt_before plus a Slack button or webhook for approve or reject.
Illustrative: refund of $4,200 trips the auto-approve limit → 1 interrupt() raises GraphInterrupt, surfaces amount → 2 state snapshot written to Postgres → 3 worker process exits, nothing idles → 4 manager taps Approve in Slack → 5 graph re-invoked with same thread_id and a Command → 6 resumes from checkpoint, calls payments API once (idempotent), logs result
Idempotency keys
Durability doesn't stop a step running twice
What it is. A safeguard, built into the external call itself, that lets a receiving system recognise and drop a duplicate request.
Why it exists. If a process dies after the payments API call succeeds but before the result is saved, resume will retry that step and call the API again. Checkpointing alone cannot know whether the first call already succeeded.
Where it breaks. This is not something the orchestration framework can enforce — it has to be designed into every external call the agent makes.
How that's handled. Attach a unique key to each side effect so the downstream system (a payments API, a CRM) treats a repeat as a no-op.
In technical terms
In technical terms: LangGraph's own documentation warns that a task which started but didn't finish may run again, so side effects must be designed to be idempotent.
Compared with what came before
| Restart from scratch | Durable execution | |
|---|---|---|
| Recovery unit | Entire run | The step that was in flight |
| Where state lives | Inside the process | External database, keyed by run identifier |
| Human approval | Holds a process or thread open | Parks the run, resumes on re-invoke |
| Duplicate side effects | Also a risk, just less visible | Still a risk — must be handled separately |
| Latency cost | None, until the crash | Depends on durability mode chosen |
Durable execution buys recoverability, not correctness. A run that resumes cleanly can still call the payments API twice if nobody added an idempotency key; the framework only guarantees that progress isn't lost, never that an external action happened exactly once.
Where it breaks
Not full workflow orchestration
LangGraph's checkpointing is lighter-weight fault tolerance within a single agent graph, not cross-process workflow orchestration — dedicated engines like Temporal, Inngest or DBOS exist for that broader job.
A workflow re-entering a step after a crash is not the same as a workflow that replays its whole history. Journal-and-replay systems — LangGraph's own Functional API, Temporal, Inngest, DBOS — replay the handler from the start and return saved results for steps already finished, rather than re-entering mid-graph. Mixing up these two models when reasoning about a crash can lead to wrong assumptions about what gets re-executed.
ForgeFlow's own repository is candid about the gap between infrastructure and safety: its support_ops and finance_recon workflows are flagged as "honest scaffolds" that raise an error on .run() unless dry_run=True or an environment flag is set, because they are templates rather than fully wired production paths. Durable plumbing existing is not the same as a workflow being safe to run unattended.
The sharpest failure mode is the one idempotency keys exist to prevent: a process crashing after the payments API call succeeds but before the result is saved, so resume retries the step and fires the call again. Durability guarantees the run picks back up; it says nothing about whether the thing the run was doing out in the world already happened.
Numbers to carry
| Figure | Value |
|---|---|
| Durability mode PR #5771 merged | 2025-08-01 |
| Durability flag PR #5963 (Python SDK) merged | 2025-08-27 |
| Durability modes available | sync, async, exit |
| Companies cited by LangChain as running LangGraph in production | Klarna, Uber, J.P. Morgan |
| ForgeFlow checkpoint backend | PostgreSQL |
Myths worth killing
| Commonly repeated | What is actually true |
|---|---|
| "Durable execution means the agent can't lose anything" | It means the agent's recorded state survives a crash; an in-flight external call can still be repeated |
| "Checkpointing and journal-and-replay are the same idea" | Checkpoint-and-resume re-enters mid-graph; journal-and-replay reruns the handler from the top, returning saved results for finished steps |
| "interrupt() just pauses a thread" | It raises a GraphInterrupt, saves state, and frees the process; nothing stays blocked waiting |
| "If it's durable, it's production-ready" | ForgeFlow's own scaffolded workflows refuse to run outside dry_run mode, showing durable plumbing and production safety are separate questions |
Worked example
- The refund agent qualifies a refund request, finishing step one cleanly; this gets checkpointed to Postgres under its run identifier.
- The agent calls the payments API to issue the refund — illustrative amount, ₹4,200 — and the call succeeds.
- The server is redeployed mid-step, before the result of that call is saved; the process dies.
- A new worker, running the redeployed code, resumes the run by its stored identifier and finds the last checkpoint sits before the payments call.
- The framework retries that step, which means calling the payments API again with the same illustrative amount.
- The payments API, because the first call carried an idempotency key, recognises the repeat and returns the original result instead of refunding a second time.
- The agent logs the result and, since the amount exceeds a threshold, calls
interrupt()to pause for an approver. - An approver, three hours later, clicks approve in Slack; the graph resumes with that decision and closes out the run.
Should you use it?
Durable execution earns its complexity when a run involves paid or stateful external calls that must not be silently repeated on crash, when a workflow must pause for approval that might arrive days later without holding a process open, or when a redeploy could plausibly land mid-run. It is overkill for a short, single-call interaction with no multi-step state, or for anything where restarting a failed run from scratch is cheap and has no duplicate-side-effect risk. The cheapest first experiment is to checkpoint one existing multi-step agent to Postgres and deliberately kill the process mid-run to see what, if anything, gets repeated.
Say it in the room
"Checkpoint anything that calls a paid API or waits on a human, and put an idempotency key on every external call before you trust the resume."
Lines to carry
- "Durability saves progress, not promises about what already happened."
- "A checkpoint tells you where you got to, not what you did out there."
- "Resuming is cheap; knowing if the last step finished is the hard part."
- "Pause the workflow, not the process."
- "Durable plumbing existing is not the same as a workflow being safe to run unattended."
FAQ
Does durable execution stop a payment from being made twice?
No, not by itself. It guarantees the workflow's recorded progress survives a crash and can resume; whether an external call like a payment happens once depends on an idempotency key attached to that call, not on the orchestration framework.
Is LangGraph's checkpointing the same thing as Temporal or Inngest?
No. LangGraph's checkpointing is described as a lighter-weight mechanism for fault tolerance and human-in-the-loop within a single agent graph, re-entering mid-graph after a crash. Temporal, Inngest and DBOS are journal-and-replay engines that replay the whole handler from the start, returning saved results for steps already finished.
What happens while a workflow is waiting for human approval?
Nothing is held open. Calling interrupt() saves the workflow's state and frees the process; the run only continues when it is re-invoked later with the human's decision attached, which could be hours or days afterwards.
Which durability mode should I pick?
Match it to the stakes. durability="sync" persists state before the next step starts and is the safest, if slowest, choice — appropriate for anything touching money or irreversible actions. durability="exit" does not recover from a mid-execution crash at all, which is fine only where losing a run costs nothing.
Is checkpointing infrastructure enough to call a workflow production-ready?
Not on its own. ForgeFlow's own project flags some of its workflows as scaffolds that refuse to run outside a dry-run mode, precisely because durable checkpointing was wired up before the workflow itself was judged safe to run unattended.
Glossary
- Durable execution — storing an agent run's progress outside the process that runs it, so a crash, deploy, or long wait can be resumed instead of restarted.
- Checkpoint — a saved snapshot of a workflow's state at a point in time, usually stored in a database, used to resume after a failure.
- Super-step — in LangGraph, a single tick of the graph where all nodes scheduled to run at that point execute, potentially in parallel.
- Journal-and-replay — a durability approach where an engine records completed step results and, after a crash, replays the handler from the start, reusing recorded results instead of re-running finished steps.
- Idempotency key — a unique identifier attached to a request so that if it is sent more than once, the receiving system processes it only once.
- Interrupt (LangGraph) — a function call inside a node that pauses the graph, saves its state, and waits for external input before continuing.
- Thread ID — an identifier used by LangGraph to track a specific run, so resuming with the same identifier reloads that run's last saved checkpoint.
- StateGraph — LangGraph's core data structure representing a workflow as nodes and edges operating on a shared state object.
- Human-in-the-loop (HITL) — designing a workflow so a person can inspect, approve, or modify its state at a defined point before it continues.
- Orchestration framework — software that manages the order, state and error-handling of multiple steps or agents, as opposed to the AI models doing the reasoning.
Sources
- LangGraph overview - Docs by LangChain — LangChain
- JoelJohnsonThomas/ForgeFlow — GitHub
- Durable Execution for AI Agents: Checkpoints, Retries and Resume — Developers Digest, 2026
- Durable Execution for Long-Running Agents — ChangeGamer — ChangeGamer
- feat(langgraph): add durability mode for
invokeandainvoke· Pull Request #5771 — GitHub, 2025 - feat(sdk-py): add durability flag · Pull Request #5963 — GitHub, 2025