DM Explains · No. 001 · Orchestration
Workflow stitching: why AI orchestration that scores 92% still fails 100% in production
Chaining agents and tools into one pipeline sounds like the obvious next step after single-shot AI. The gap between benchmark quality and executable output is where it breaks.
· 13 min read
Read me first
Chaining agents and tools into one pipeline genuinely does let a system handle tasks no single model can, but only once the workflow itself can be checked by machine — reasoning quality was never the bottleneck.
Refund request pipeline: customer message → 1 split into classify/policy/decide → 2 DAG built over 25 components → 3 88%-accurate selection of classifier, policy-lookup, decision agent → 4 each gets only its task, not full chat → 5 schema-constrained JSON checked, malformed output repaired → 6 approved refund logged with reasoning trail for audit
A team wires together a classifier, a policy lookup and a refund-decision agent, tells the model to figure out the routing itself, and demos it beautifully on ten test intents. Everyone nods. It ships.
In deployment, across 1,200 runs, the same design failed before meaningful execution every single time — not because the model reasoned badly, but because its free-form plan could not be parsed into a workflow a machine could actually run. The gap between "sounds right" and "runs" is where workflow stitching lives or dies.
Everyday picture. Amma plans dinner by telling Arjun to chop vegetables, Papa to light the stove and Meera to set the table, then checks each person's work before anything reaches the table.
Where the picture breaks. Amma can see instantly if Arjun's chopping is wrong; an AI orchestrator often has no eyes on a step's output beyond an explicit schema check, which is exactly why unchecked systems fail silently instead of visibly.
Why it exists now
Through 2026, orchestration frameworks and enterprise gateways pushed "dynamic intent-to-workflow" orchestration into production: a plain-language request goes in, an executable workflow graph comes out. The pitch is obvious — instead of hand-coding every pipeline, let the system design the pipeline itself, per request.
A 2026 industry deployment study, presented at the ASE 2026 industry track, tested this directly. A system built over 25 registered components was run across 50 intents and 1,200 runs. In controlled benchmarking it scored well: 80.63% structural similarity to expert-designed workflows, 88% component-selection accuracy, and 92% of manual-workflow output quality (ASE 2026 industry track, September 2026). Then, in real deployment, every automated orchestration attempt failed before meaningful workflow execution — because free-form model outputs could not consistently be turned into machine-valid workflow specifications (ASE 2026 industry track, September 2026).
That single result is why orchestration design changed in 2026: benchmarks measure whether the reasoning was good; production measures whether the output was executable. Those are different tests, and a system can pass one and fail the other completely.
How it works
Intent decomposition
No plan, no basis for choosing anything
What it is. A high-level ask — "handle this refund request" — is translated into a plan naming subtasks (classify, look up policy, decide) and how they depend on each other.
Why it exists. Orchestration is only needed once a task requires more than one kind of judgement, coordinated across separate agents or tools; decomposition is what turns one vague ask into several narrow, assignable jobs.
Where it breaks. If decomposition is vague, every downstream step inherits the ambiguity — a fuzzy plan cannot be tightened by good execution later.
How that's handled. Treat the decomposition step as a checkpoint in its own right, not a throwaway prompt; log it so failures can be traced back to it.
In technical terms
In technical terms: orchestration handles task routing, model and agent selection, context and state management, tool execution, output validation, error recovery, human escalation and audit logging as distinct responsibilities, starting with decomposition.
Graph construction
A plan without a graph can't be checked
What it is. The plan becomes a directed graph — nodes for steps, edges for dependencies — so cost, order and retries can be reasoned about before anything runs.
Why it exists. A graph representation is what makes a workflow inspectable. Without it, "the plan" is just a paragraph of text that no software can validate.
Where it breaks. A graph only helps if it's actually checked; skipping validation just moves the failure downstream to execution time, where it's harder to diagnose.
How that's handled. The studied deployment built its DAGs (directed acyclic graphs) over 25 registered components, which let structural similarity to expert workflows be measured directly — 80.63% in controlled benchmarking (ASE 2026 industry track, September 2026) — rather than guessed at.
In technical terms
In technical terms: a DAG has nodes selecting tools or agents and edges encoding data dependencies between them, letting the orchestrator validate the shape of a workflow before or during execution rather than discovering problems mid-run.
Component selection
Right tool beats clever tool
What it is. For each node in the graph, the orchestrator picks the specific model, agent or API best suited to that subtask from a registry of available components.
Why it exists. Once the workflow's shape is valid, the single biggest lever on output quality is choosing the right component for each job — not making any one component cleverer.
Where it breaks. Good selection on an invalid graph still produces nothing executable; selection quality cannot compensate for a topology that doesn't parse.
How that's handled. The same studied system reached 88% component-selection accuracy against expert choices across the 50 intents (ASE 2026 industry track, September 2026), measured separately from whether the overall workflow ran.
In technical terms
In technical terms: component selection is scored against expert-chosen baselines per node, isolating "did it pick the right tool" from "did the graph itself work."
Handoff and state
Pass the task, not the whole conversation
What it is. Each step receives only the task, relevant context, constraints and expected output format — never the full history of everything that came before.
Why it exists. Passing an agent the entire conversation raises cost and degrades output; a well-designed handoff includes the specific task, relevant context, constraints and a clear expected format, and nothing else.
Where it breaks. Trim too aggressively and a detail the next agent actually needed never arrives, producing a confidently wrong output instead of an obviously broken one.
How that's handled. The supervisor pattern — one orchestrating agent or deterministic logic receiving the overall task, splitting it, routing pieces to specialists, and assembling results — is described as the most common and most reliable design in production multi-agent systems.
In technical terms
In technical terms: the supervisor pattern routes subtasks to specialist agents and assembles their outputs, with each handoff deliberately scoped rather than carrying the accumulated conversation history.
Validate and repair
Free text breaks; schemas don't
What it is. Outputs are checked against a schema; malformed or partially valid workflow specifications are retried or repaired using function calling, tool-use APIs or JSON mode rather than parsed as free text.
Why it exists. This is the step that decided the ASE study's outcome, and it's the one most teams skip. Free-form model outputs are too fragile to be parsed reliably into executable workflows without it.
Where it breaks. Without this layer, benchmark scores of 80.63% to 92% mean nothing operationally — the ASE 2026 study found every automated attempt failed before meaningful execution in real deployment (ASE 2026 industry track, September 2026), despite those same scores in controlled testing.
How that's handled. The paper's guidance: prefer function calling, tool-use APIs, schema-constrained generation or JSON modes whenever the workflow representation itself must be executable, and adopt a template-first rollout, reserving dynamic orchestration for novel, higher-complexity tasks where fixed templates give poor coverage.
In technical terms
In technical terms: schema-constrained generation forces the model's output into a fixed structure a validator can check mechanically, replacing free-text parsing with pass/fail structural checks and automatic repair loops.
Escalate and log
Someone must be able to ask why
What it is. Cases the system cannot resolve are flagged for a human, and every routing, selection and execution decision across the pipeline is recorded.
Why it exists. Orchestration explicitly includes human escalation and audit logging as core responsibilities, alongside routing and validation — not as an afterthought bolted on for compliance.
Where it breaks. Logging after the fact doesn't recover a workflow that already failed silently; it only shortens the time to find out why.
How that's handled. Trace failures to the specific step — decomposition, graph, selection, handoff or validation — rather than treating a bad output as one undifferentiated mystery.
In technical terms
In technical terms: audit logging captures the decision made at each node (which component was selected, what was passed, what validation returned) so a failed run can be attributed to a single point of the graph.
Compared with what came before
| Template-based pipeline | Dynamic workflow stitching | |
|---|---|---|
| Structure | Fixed, versioned sequence of steps | Graph built fresh from the intent, per run |
| Predictability | High — bounded cost and behaviour | Lower — depends on validation layer |
| Coverage | Only what was templated in advance | Handles novel or open-ended intents |
| Failure mode | Doesn't attempt unfamiliar tasks | Can produce an unparseable plan that never executes |
| Best measured result | N/A, by design | 80.63% structural similarity, 88% component-selection accuracy, 92% of manual-workflow output quality (ASE 2026 industry track, September 2026) |
Dynamic orchestration buys flexibility that templates cannot offer, but it gives up the predictability that made templates safe to deploy unattended. The ASE 2026 industry study's own conclusion was to keep templates as the default and reserve dynamic orchestration for cases where template coverage is genuinely poor — not to replace templates outright.
Where it breaks
Workflow stitching earns its place when a task genuinely needs more than one kind of judgement chained together, and it fails hardest exactly where teams assume benchmark quality is the whole story.
Benchmark quality is not deployment quality
A system scoring 88% component-selection accuracy and 92% of manual-workflow output quality in controlled testing still failed before meaningful execution in every deployment run tested, because those scores measure reasoning quality on a workflow that was assumed to be valid — not whether the workflow could actually be parsed and run (ASE 2026 industry track, September 2026).
The conditional matters here: this is not "orchestration doesn't work", it's "orchestration doesn't work without a validation and repair layer that most teams haven't built yet." Cost is the second failure mode — per-token pricing projections routinely ignore the cost of retries, validation and the integration engineering needed to wire a registry of tools together in the first place. And graph engineering — pinning the topology to a fixed, versioned file rather than letting the model improvise it fresh each run — trades away flexibility for exactly the predictability a template gives, which is often the right trade for routine, high-frequency tasks.
Numbers to carry
| Metric | Value |
|---|---|
| Registered components in the studied system | 25 (ASE 2026 industry track, September 2026) |
| Intents / runs tested | 50 intents, 1,200 runs (ASE 2026 industry track, September 2026) |
| Structural similarity to expert workflows | 80.63% (ASE 2026 industry track, September 2026) |
| Component-selection accuracy | 88% (ASE 2026 industry track, September 2026) |
| Manual-workflow output quality reached | 92% (ASE 2026 industry track, September 2026) |
| Real-deployment execution failure rate | Every attempt failed before meaningful execution (ASE 2026 industry track, September 2026) |
Myths worth killing
| Commonly repeated | What is actually true |
|---|---|
| "If the benchmark score is high, it will work in production." | The ASE 2026 study scored 80.63% to 92% across three benchmarks yet failed before meaningful execution in every deployment run, because benchmarks tested reasoning, not machine-valid output. |
| "One smart agent can just handle the whole workflow." | Orchestration is needed once a task requires more than one kind of judgement — classification, policy lookup and decision are three different jobs, not one. |
| "More context passed between agents is always safer." | Handoffs are deliberately scoped to the task, relevant context, constraints and expected format; passing the full history raises cost and degrades output. |
| "Dynamic orchestration should replace fixed templates." | The study's own recommendation is a template-first rollout, with dynamic orchestration reserved for novel, high-complexity tasks where templates give poor coverage. |
Worked example
- Amma (standing in for the customer-facing system) receives a message: "I want a refund for my broken order."
- The orchestrator decomposes the intent into three subtasks: classify the request, look up refund policy for this product category, decide approve or deny.
- The orchestrator builds a DAG connecting the three subtasks in sequence, with the policy lookup feeding into the decision step.
- The orchestrator selects a classifier agent, a policy-retrieval tool and a decision agent from its component registry for the three nodes.
- The classifier agent receives only the customer message and returns a category — illustrative: "damaged item, category electronics" — nothing else is passed forward except this trimmed result.
- The policy-retrieval tool receives the category, not the original message, and returns the applicable refund window — illustrative: "30 days, electronics."
- The decision agent receives the category and the policy result, checked against a schema before being accepted, and returns approve or deny in a fixed structured format.
- The orchestrator validates that structured output; if it fails validation, it retries the decision step with a repair prompt rather than passing broken output onward.
- The orchestrator logs every step's input, output and validation result, and escalates to a human reviewer only if the decision agent's output still fails validation after repair attempts.
Should you use it?
Use dynamic workflow stitching when the task genuinely needs more than one kind of judgement, there's a real registry of tools or agents to choose between, and the organisation is willing to build schema-constrained validation before trusting any of it. Skip it — or start with fixed templates instead — for routine, high-frequency tasks where a deterministic pipeline already gives predictable, bounded-cost execution, and for any production setting where nobody has built the validation and repair layer yet.
The cheapest first experiment is not a new pipeline: it's taking an existing script-glued pipeline and adding a schema check between just two of its steps, then measuring how often that check catches something the demo never revealed.
Say it in the room
"Benchmark scores tell you the reasoning was good; they tell you nothing about whether the output could be parsed into something that runs."
Lines to carry
- "A workflow that reasons well but can't be parsed doesn't run at all."
- "Pass the task, not the whole conversation."
- "Benchmarks measure reasoning; production measures machine-valid output."
- "Fix the topology before you trust the improvisation."
- "A graph you can check beats a plan you can only hope worked."
FAQ
Isn't a high benchmark score enough to trust a system in production?
No — the ASE 2026 industry study is the clearest evidence against that. The same system that scored 80.63% structural similarity, 88% component-selection accuracy and 92% of manual-workflow output quality in controlled testing failed before meaningful execution in every one of its deployment runs, because benchmark scores were measuring reasoning quality on workflows already assumed to be valid, not whether they could actually run.
What's the difference between one agent using tools and real orchestration?
One agent completing a task end-to-end, even if it calls a few tools along the way, isn't orchestration in the sense that matters here. Orchestration exists once a workflow needs more than one kind of judgement — classification, policy lookup, and a decision, say — coordinated across genuinely separate agents or components.
Why not just let the model figure out the whole plan in free text?
Because free text can't be checked by a machine before it's run. That's exactly the failure mode the ASE 2026 study identified: free-form outputs couldn't be reliably turned into machine-valid workflow specifications, which is why the recommendation is to force outputs through function calling, tool-use APIs or JSON mode wherever the output has to be executable.
Should every AI pipeline use dynamic orchestration instead of fixed templates?
No. The same study's own guidance is a template-first rollout — start with fixed, predictable templates and reserve dynamic orchestration for tasks that are genuinely novel or complex enough that template coverage is poor. Templates give up flexibility but keep the predictability that dynamic orchestration currently struggles to guarantee.
How do agents avoid confusing each other when handing off a task?
By trimming what's passed. A well-designed handoff carries the specific task, relevant context, constraints and the expected output format — not the entire conversation history up to that point. Passing everything raises cost and tends to degrade the next agent's output rather than improve it.
Glossary
- Orchestration — coordinating multiple AI models, agents and tools so they work together towards one goal, rather than running independently.
- Agent — a software component, often powered by an AI model, that can take actions such as calling tools or APIs to accomplish a narrow task.
- DAG (directed acyclic graph) — a diagram of steps and their dependencies with no loops, used to represent the order in which parts of a workflow must run.
- Schema-constrained generation — forcing an AI model's output to follow a fixed structure so software can reliably read it.
- Tool use / function calling — a model producing a structured request to call an external tool or API rather than free-form text.
- Supervisor pattern — a design where one orchestrating agent or deterministic logic receives the overall task, splits it up, and routes pieces to specialist agents.
- Component-selection accuracy — how often the orchestration system picks the correct tool, model or agent for a subtask, measured against expert choices.
- Structural similarity — a measure of how closely an automatically generated workflow's shape matches one designed by a human expert.
- Graph engineering — fixing a multi-agent system's coordination structure into a stable, versioned topology rather than generating it fresh at run time.
- Template-first rollout — a deployment approach that starts with fixed, predictable automation templates before introducing dynamically generated workflows.
- Handoff — the point at which one agent passes a task, context, constraints and expected format to another agent in a workflow.
- Human-in-the-loop — a step where a human reviews or approves an AI system's output before the workflow continues.
Sources
- Dynamic AI Workflow Orchestration in Practice: Deployment Lessons — ASE 2026 Industry Track, 2026