DM Explains · No. 007 · Orchestration
Checkpointed workflows: why your AI agent shouldn't start over after a crash
Long-running agents chain dozens of LLM calls and tool actions together. Without saving progress after each step, one crash or one paused approval means paying for and repeating all of it.
· 13 min read
Read me first
Checkpointed workflows genuinely do save an agent from repeating expensive work after a crash, but only when something else is watching for the crash in the first place — the checkpoint itself is just a memory, not a reflex.
Illustrative: agent researches a lead → 1 each node's state persisted to Postgres → 2 run ID + last node ID stored so any worker can resume → 3 draft email pauses for Slack approval → 4 worker crashes mid-send; restart detects the dead process → 5 resume skips research and drafting, already done → 6 replay reconstructs state from the append-only history → 7 CRM write fails, prior steps roll back LIFO → email resent, outcome logged exactly once
An agent workflow has been running for twenty minutes: three calls to a large language model, a database lookup, a drafted follow-up email now sitting and waiting for a human to approve it. The server redeploys mid-run.
Without saved progress, the whole chain restarts from step one. The three LLM calls get billed again. Worse, if the drafted email had already gone out before the crash, the retry might send it a second time. Twenty minutes of orchestration, and the only proof it ever ran is a duplicate email in someone's inbox.
Everyday picture. Amma is cooking a multi-step dinner — chopped, marinated, fried — and after each stage she writes it on a card stuck to the fridge instead of trusting her memory. If she steps out mid-cook, Papa can check the card and continue from exactly that stage, rather than starting the marinade over.
Where the picture breaks. The fridge card doesn't itself notice if Amma never comes back. Someone still has to check it and decide to carry on cooking — that watching-and-deciding step is exactly what a plain checkpoint is missing.
Why it exists now
Agentic AI workflows chain many LLM calls and tool invocations together over minutes or hours, not seconds. A single crash, a flaky API call, or a pause for human sign-off used to mean a full re-run — repeating billed model calls and, worse, repeating side effects like emails or CRM writes that shouldn't happen twice.
Through 2025 and 2026, open-source agent orchestration frameworks converged on checkpoint-and-resume as a standard feature for enterprise deployment: ForgeFlow, built on LangGraph with PostgreSQL checkpointing (JoelJohnsonThomas/ForgeFlow — GitHub), AgentFlow4J's checkpoint-and-resume tutorial for Spring Boot agents (Checkpoint and resume agent workflows in Spring Boot — AgentFlow4J), and Duraflow's saga-pattern rollback (Harsh-Vaghela-404/duraflow — GitHub). Diagrid's write-up on durable execution argued the sharper point: checkpointing alone is not enough — something still has to notice a process died and decide to reload it (Durable Execution: The Missing Runtime Primitive for Agents — Diagrid). That argument is pushing wider adoption of durable-execution runtimes built specifically for AI agents.
How it works
Step-level persist
Nothing survives that isn't written down
What it is. After each step in the agent's graph completes, the full state — variables, tool outputs, conversation history — is serialised and written to durable storage.
Why it exists. State kept only in a process's memory disappears the instant the process dies. Writing it out after every step means a crash loses, at most, the work done since the last write.
Where it breaks. Writing state after every single step adds latency; that matters when a step needs a sub-second response.
How that's handled. Teams accept the small write cost per step in exchange for never losing a completed run to a crash.
In technical terms
In technical terms: ForgeFlow persists every node to PostgreSQL through a LangGraph StateGraph, and a separate pattern write-up describes serialising the entire workflow state — step index, local state, awaited signals, history — into a pluggable storage provider such as Postgres, S3 or Redis.
Run + node ID
Anyone can pick up where the story stopped
What it is. Every checkpoint carries a stable run identifier and the name of the last completed step, not just raw data.
Why it exists. Without an identity attached to the state, no other process could find it or know where to continue.
Where it breaks. Identity tracking only helps if workers are interchangeable; if a step depends on something tied to one specific machine, another worker resuming it will fail anyway.
How that's handled. Workflows are designed so any worker can pick up any run by ID — no worker owns a run permanently.
In technical terms
In technical terms: AgentFlow4J's checkpoint stores a run ID that stays stable across restarts, plus the name of the last completed node, so calling the resume function with that run ID from any process skips straight to the next unfinished step.
Human gate
Pause before the point of no return
What it is. Before a high-impact action — sending an email, writing to the CRM — the workflow stops and waits for a person to approve or reject it.
Why it exists. Some actions can't be undone once fired; a computer program deciding alone to send a customer email is a risk most teams won't accept.
Where it breaks. An indefinite pause is only safe if the process is fully checkpointed and torn down while waiting, not merely left idling in memory, which would cost compute for no reason.
How that's handled. The pause is implemented as a stop-and-reload point, resumed later by an external signal, not a live thread waiting around.
In technical terms
In technical terms: ForgeFlow halts execution using LangGraph's interrupt_before construct before a node runs, then resumes on an approve or reject webhook or a Slack button; a fork of the project adds three modes — inline, checkpoint, and auto-escalate with a configurable timeout.
Illustrative: CRM agent drafts a follow-up → 2 state serialized to Postgres → 3 process torn down entirely, worker exits → 4 sales rep reviews for 3 days → 5 checkpoint reloaded, no re-research, no re-draft → 6 email sent and outcome logged to the CRM
Failure detection
A checkpoint alone does nothing until someone looks
What it is. A separate mechanism that notices a process has died or a run has stalled, and explicitly triggers reloading the last saved checkpoint.
Why it exists. A stored checkpoint is inert data. Nothing resumes automatically unless something is watching for failure.
Where it breaks. Teams that add periodic state-saving without this watcher layer believe they have durable execution but do not — the state sits in the database, unread, forever.
How that's handled. Durable execution runtimes add exactly this layer as their central job, separate from the checkpoint store itself.
In technical terms
In technical terms: as Diagrid's write-up puts it, a retry re-invokes the same failed step but does nothing for steps completed earlier in the run; a checkpoint saves state but still needs a runtime that notices the process died and reloads it.
Skip on resume
Finished work stays finished
What it is. On resume, steps already completed are not run again — their stored results are returned instead.
Why it exists. Re-running a completed step would bill the LLM call twice and could repeat a side effect, like sending the same email a second time.
Where it breaks. This assumes a step's output is safe to reuse rather than re-fetch; a step that reads live external state (a stock price, an inventory count) may return a stale, cached answer instead of a current one.
How that's handled. Frameworks memoise step outputs by default and leave it to the workflow author to mark any step that genuinely must re-run.
In technical terms
In technical terms: AgentFlow4J skips completed nodes entirely on resume, avoiding repeated LLM calls or duplicate tool invocations, and Duraflow memoises completed step results so a retried step returns cached output rather than triggering duplicate LLM billing.
Replay history
The full trail, not just the last snapshot
What it is. Instead of storing only the latest snapshot of state, some runtimes keep an append-only log of every step and rebuild current state by replaying that log.
Why it exists. A full history gives an auditable trail of exactly what happened, useful for compliance and debugging, beyond just knowing the current state.
Where it breaks. Replay only reconstructs correctly if the step logic is deterministic; a step that reads the system clock or a random number will reconstruct differently each time.
How that's handled. Workflow authors are expected to keep step logic deterministic and to record any external randomness as data, not recompute it on replay.
In technical terms
In technical terms: durable execution runtimes such as Dapr use event-sourcing replay, writing an append-only history of every step and reconstructing state on resume by replaying it, with completed activities returning their stored results instead of re-executing.
Saga rollback
Undoing is as designed-for as doing
What it is. When a later step in the workflow fails, earlier completed steps are automatically undone, in reverse order.
Why it exists. Forward progress alone isn't enough in production — if a workflow can't finish, partially completed side effects (a partial CRM write, a reserved resource) need a defined way to be unwound.
Where it breaks. Rollback only works for actions that can genuinely be compensated; an email already read by a human cannot be un-sent or un-read.
How that's handled. Failed rollbacks are routed to a separate queue for manual handling rather than silently dropped.
In technical terms
In technical terms: Duraflow rolls back prior steps in LIFO order following the saga pattern when a later step fails, and routes any compensation that itself fails to a dead-letter queue.
Compared with what came before
| Plain retry | Checkpointed workflow | |
|---|---|---|
| On crash | Re-invokes the failed step; earlier completed steps are simply gone | Reloads the last checkpoint; only unfinished work is redone |
| Human pause | Process must stay alive, waiting | Process is torn down; resumes later on a signal |
| Duplicate side effects | Likely on any re-run | Skipped via memoised step outputs |
| Failure detection | Assumed to be handled elsewhere | Still a separate concern, not solved by the checkpoint itself |
| Audit trail | Usually none | Some runtimes keep a full replayable history |
The judgement is not that checkpointing replaces retries — a retry still handles the failed step itself. Checkpointing is what makes retrying cheap: it decides how much of the run has to be redone, and the honest answer, when it's set up properly, is "just the unfinished part."
Where it breaks
Checkpointed workflows earn their keep when a run genuinely spans multiple expensive steps with side effects worth protecting; they add pure overhead when a task is a single stateless call with nothing to lose.
A checkpoint is not a runtime
Saving state periodically does nothing on its own. Something has to notice the process died and explicitly reload the saved state — this is the argument Diagrid makes directly, and it's the gap between "we save state" and "we have durable execution."
Replay-based reconstruction depends on deterministic step logic; any step that reads a clock or generates a random value can reconstruct differently on replay, silently corrupting the history it's meant to preserve. Saga rollback depends on actions being compensable at all — some side effects, once observed by a human or another system, cannot be truly undone regardless of how carefully the rollback is designed. And step-level persistence, however cheap per write, is still a cost paid on every single node; workflows that need sub-second responses on every step will feel that overhead directly.
Numbers to carry
| Detail | Value |
|---|---|
| ForgeFlow checkpoint backend | PostgreSQL, via a LangGraph StateGraph |
| ForgeFlow pause mechanism | LangGraph's interrupt_before, resumed via approve/reject webhook or Slack button |
| Human-in-the-loop modes (MatthewLones' ForgeFlow fork) | three modes: inline, checkpoint, auto-escalate (with configurable timeout) |
| checkpointflow default storage path | ~/.checkpointflow/ |
| Duraflow rollback order | LIFO (last completed step undone first), following the saga pattern |
| Transact checkpoint points | before execution (inputs) and after each step (outputs), per lockitsoft.com's architecture write-up |
Myths worth killing
| Commonly repeated | What is actually true |
|---|---|
| "Checkpointing means the workflow resumes itself" | A checkpoint is saved data; a separate watcher must detect failure and trigger the reload, as Diagrid's write-up argues |
| "Retrying and checkpointing solve the same problem" | A retry redoes only the failed step and does nothing for steps completed earlier; checkpointing is what preserves that earlier progress |
| "Pausing for human approval means the process just waits" | In frameworks like ForgeFlow, the process is torn down entirely at the interrupt point and reconstructed later from the checkpoint, not left idling |
| "Once state is saved, resuming always redoes nothing extra" | Only steps proven safe to skip are memoised; steps reading live external data can return stale cached results if reused blindly |
| "Rollback undoes any side effect" | Rollback only works for compensable actions; some effects, like an email already read, cannot be undone, which is why failed compensations go to a dead-letter queue in Duraflow |
Worked example
- The CRM lead-qualification agent starts a run: it researches a lead, drafts a follow-up email, and writes progress after each step.
- Step one (research) completes; the agent's context — the lead's details, the research findings — is serialised to durable storage along with the run's ID and the name "research" as the last completed node.
- Step two (draft email) completes; the draft and updated state are persisted the same way.
- The workflow pauses before sending, using a human-gate interrupt; the process is torn down rather than left running, waiting for a person to click approve.
- The server redeploys at this exact point — illustrative, but plausible given the twenty-minute run described earlier. No process is currently running the workflow at all.
- A watcher process notices the run is stalled (or, more precisely here, correctly paused) and, once the approval webhook arrives, reloads the last checkpoint using the run ID.
- Resuming, the workflow skips the research and drafting steps entirely — their results are read from storage, not recomputed — and proceeds straight to sending the approved email.
- The email is sent once, the outcome is logged to the CRM, and the two earlier LLM calls are never billed a second time.
Should you use it?
Checkpointing earns its cost when a workflow chains multiple LLM calls or tool actions long enough that redoing them from scratch would be expensive, when a human approval must pause execution for an indefinite stretch, when several worker processes need to pick up and resume any run interchangeably, or when the workflow performs side effects — emails, CRM writes, payments — where accidental duplication would cause real harm.
It's not worth the overhead for a single-turn, stateless call that finishes in one request-response cycle, for a workflow cheap enough to simply rerun from the start with no side effects worth protecting, or for a step that genuinely needs sub-second latency on every hop. The cheapest first experiment is not a rewrite: pick one existing multi-step agent, add persistence after each of its steps using something already in the stack like Postgres, and confirm a mid-run kill can be resumed from the last completed step rather than the first.
Say it in the room
"Checkpoint every step that costs money or has a side effect; the crash isn't the risk, redoing finished work is."
Lines to carry
- "A checkpoint is a memory, not a reflex."
- "Retrying redoes the failed step; checkpointing keeps everything before it."
- "Finished work should stay finished."
- "Undoing needs as much design as doing."
- "A saved checkpoint nobody watches is just a file."
FAQ
Isn't this just retry logic with extra steps?
No — a retry re-invokes the failed step and does nothing for everything completed earlier in the run. Checkpointing is specifically what preserves that earlier progress, so a retry only has to redo the small unfinished part, not the whole run.
Do I need a dedicated durable-execution runtime, or can I just save state myself?
You can save state yourself, and several frameworks (ForgeFlow, AgentFlow4J, Duraflow) show how. But saved state alone doesn't resume anything — something still has to detect the crash and reload it, which is exactly the gap Diagrid's write-up on durable execution calls out. If nothing is watching, you have a checkpoint file, not a resumable workflow.
What happens if two steps run at once and one fails?
That depends on whether the framework supports saga-style rollback. Duraflow, for one, undoes previously completed steps in reverse order when a later step fails, and sends any rollback that itself fails to a separate queue rather than losing track of it.
Does checkpointing slow down every step?
Yes, some — writing state to durable storage after each node adds latency compared with keeping everything in memory. For workflows needing sub-second responses at every step, that overhead is a real cost to weigh against the protection it buys.
Can I checkpoint a workflow that reads live external data, like a stock price?
Carefully. Memoisation on resume assumes a step's stored output is safe to reuse instead of re-fetched. If a step depends on data that changes over time, skipping it on resume can return a stale answer rather than the current one — that step may need to be excluded from memoisation deliberately.
Glossary
- Checkpoint — a saved snapshot of a workflow's state at a point in time, written to durable storage so it can be reloaded later.
- Durable execution — an execution model where every completed step is persisted, so a workflow can be reconstructed and resumed from the last completed step after a crash or restart.
- LangGraph — a framework for building agent workflows as a graph of nodes (steps), used to structure and checkpoint agent execution.
- interrupt_before — a LangGraph mechanism that pauses a workflow before a specified node runs, typically to wait for human approval.
- Event sourcing — a technique of storing an append-only history of every event or step, then reconstructing current state by replaying that history.
- Saga pattern — a way of handling multi-step transactions where, if a later step fails, previously completed steps are undone in reverse order via compensating actions.
- Dead-letter queue — a holding queue for failed operations, such as failed rollback actions, that need manual or separate handling rather than being silently dropped.
- Memoisation — caching the result of a step so that if it is invoked again, the stored result is returned instead of redoing the work.
- Thread ID / run ID — a stable identifier for one execution of a workflow, used to locate its checkpointed state for resumption.
- Exactly-once execution — a guarantee that a given step's side effect happens only one time even if the step's code is retried after a crash.
Sources
- JoelJohnsonThomas/ForgeFlow — GitHub
- Checkpoint and resume agent workflows in Spring Boot - AgentFlow4J — AgentFlow4J
- Durable Execution: The Missing Runtime Primitive for Agents — Diagrid
- patterns/durable-workflow-snapshot.md — GitHub
- MatthewLones/forgeflow — GitHub
- checkpointflow v1.10.0 — PyPI
- Harsh-Vaghela-404/duraflow — GitHub
- Compiling Workflows into Databases: The Architecture That Shouldn't Work (But Does) — lockitsoft.com