DM Explains · No. 025 · Orchestration
Context-aware orchestration: why agents need a context manager, not just a bigger window
Dumping the whole conversation into the model every time is not reliability, it's waste. The 2026 systems that win budget what the model sees, step by step.
· 13 min read
Read me first
Context engineering genuinely does cut wasted tokens and stale facts, but only when something decides what the model doesn't see; a bigger context window just raises the limit on how much junk can fit.
An agent swarm works across repos for 24 hours: 1 acquisition retrieves docs via RAG → 2 missing data is created by calling a test tool → 3 budgeting forces drops to fit a 6K history budget, cutting cost 10.2%-32.2% → 4 dependency tracking keeps items later steps actually reuse → 5 memory & provenance store state with its source → 6 scheduling routes work through a fixed control flow → 7 a self-managed context file is edited turn by turn, showing 65% greater improvement than prior strategies at equal compute
A coding agent left running overnight across several repositories doesn't fail because it runs out of context window. It fails because nobody decided which fact, from six hours ago, still mattered to the function being written now. The window was big enough. The decision was missing.
Context engineering is the discipline of making that decision on purpose, every step, instead of leaving it to "send everything" or "send whatever's recent."
Everyday picture. Amma packing school tiffins in the morning: she doesn't tip the whole fridge into Arjun's bag. She checks what today's timetable needs, what's left from yesterday that's still good, and packs only that.
Where the picture breaks. Amma knows why a leftover matters and can judge freshness by looking. The systems below score relevance from patterns of past use and token limits; they have no sense of why a fact matters, only signals correlated with it.
Why it exists now
The problem predates large language models: any system with a fixed working memory has to choose what enters it. What changed is that agents now run for hours, call dozens of tools, and accumulate history far past what any context window can hold cheaply. A Springer Nature Discover Artificial Intelligence perspective paper frames this directly, treating context engineering as a system-level orchestration problem sitting above prompt engineering, with acquisition, creation, persistence, assembly and budgeting as its recurring parts (Discover Artificial Intelligence, Springer Nature, 2026).
Sending the full history every call used to be tolerable when agents ran for a few turns. It stops being tolerable once an agent runs across a working day, calling tools whose outputs don't exist until the agent acts — at which point cost and dilution both grow with history length, not with relevance. That's the gap each of the following mechanisms addresses.
How it works
Illustrative: a repo-wide refactor swarm → 1 retrieves the old API docs → 2 a tool call generates a migration patch → 3 memory stores the patch for later reuse → 4 assembly scores it high on reuse, low on recency → 5 the 6K budget keeps it, drops a stale log (10.2%-32.2% cost cut) → 6 its origin is logged → delivered bounded context for the next step
Acquisition
Pull in what the model doesn't already know
What it is. Fetching external documents or facts the model wasn't trained with and placing them in context before it answers.
Why it exists. A model's training data goes stale and never contained anything private to one organisation; retrieval closes that gap at inference time.
Where it breaks. It only acquires what the retrieval step was built to find; a fact outside the indexed material simply isn't there to acquire.
How that's handled. The Springer Nature paper treats retrieval-augmented generation as exactly this: a mechanism for context acquisition, distinct from the mechanisms that follow (Discover Artificial Intelligence, Springer Nature, 2026).
In technical terms
In technical terms: acquisition mechanisms match a query against an external store and insert the top-matching passages into the prompt before generation.
Creation via tools
Some context doesn't exist until you act
What it is. Running a tool — a test suite, a search, a calculation — and feeding its output back in as new context.
Why it exists. Some facts (did the test pass? what does this file currently contain?) can't be retrieved because they don't exist until an action produces them.
Where it breaks. The output is only as current as the moment it ran; if the underlying state changes afterwards, the context is stale again.
How that's handled. The same Springer Nature paper names tool-augmented reasoning as context creation via external action, a separate mechanism from acquisition rather than a variant of it (Discover Artificial Intelligence, Springer Nature, 2026).
In technical terms
In technical terms: a tool call returns structured output that is serialised into the context alongside a record of which call produced it.
Budgeting
A token limit forces real choices, not deferrals
What it is. Scoring every candidate piece of context against a fixed token limit and keeping only what scores highest.
Why it exists. Without a budget, "send everything" is the default, and cost and dilution both grow with history length regardless of relevance.
Where it breaks. Whatever scores just below the cutoff is simply gone from that call, even if a later step needs it.
How that's handled. ContextRender selects tool results for a fixed 6K-token history budget by combining an observed-reuse signal with recency and semantic relevance, and keeps the results it omits for possible later use rather than discarding them outright (The Latest in AI, 2026).
In technical terms
In technical terms: each candidate is scored on a weighted combination of recency, semantic similarity and observed reuse, and the budget is filled greedily by score until the token limit is reached.
Dependency tracking
Track what later steps actually reuse
What it is. Recording which earlier pieces of context a later step actually drew on, so that fact can be protected from being trimmed even once it's no longer recent.
Why it exists. Recency- and similarity-based trimming can discard a fact a later step depends on simply because it's old, not because it's irrelevant.
Where it breaks. The signal is inferred from past reuse; a dependency needed for the first time has no history to score against and can still be dropped.
How that's handled. ContextRender's own evaluation, across AppWorld and an 8-objective QA benchmark with three execution models, reduces mean inference cost by 10.2% to 32.2% relative to using the full history, by weighting observed reuse alongside recency (The Latest in AI, 2026).
In technical terms
In technical terms: a dependency graph links each context fragment to the later steps observed to reference it, and that reference count is added to the scoring function used at budgeting time.
Memory & provenance
Remember state, and remember where it came from
What it is. A separate service that persists facts across turns and tags each one with where it came from.
Why it exists. Without persistence, every call starts from nothing; without provenance, a surviving fact can't be checked or traced back to its source later.
Where it breaks. Provenance has to be logged at the moment a fact enters context — it can't be reconstructed cleanly once the fact has already been trimmed or compacted.
How that's handled. The Springer Nature paper identifies memory services as the mechanism for context persistence, and separately lists provenance loss, context drift, privacy disclosure and evaluation instability as anticipated failure modes of these systems, not rare exceptions (Discover Artificial Intelligence, Springer Nature, 2026).
In technical terms
In technical terms: memory services store (fact, source, timestamp) tuples outside the model's live context and surface them back into context on later calls that need them.
Scheduling
Flexible agents inside a fixed control flow
What it is. A deterministic backbone that routes a request to the right specialist agent while that agent adjusts its own prompt or retrieval strategy locally.
Why it exists. Pure deterministic pipelines can't adapt to varied requests; pure freeform agents lose the auditability enterprises ask for.
Where it breaks. Too much fixed structure reintroduces the brittleness orchestration was meant to remove; too little loses the auditability that justified adding structure in the first place.
How that's handled. ORCHESTRA embeds large language model agents within a deterministic control flow, routing text, speech and image requests through specialist workers, while scoped agent nodes adjust prompts and retrieval strategies dynamically (ACL Anthology, 2026). Orchestra-o1 reports a related benefit from this split: modality-aware task decomposition with parallel sub-task execution surpasses the second-best approach by 10.3% accuracy on the OmniGAIA benchmark (pubdb.com, 2026).
In technical terms
In technical terms: a fixed routing graph dispatches requests to specialist workers; locally scoped agent nodes vary their internal prompt and retrieval parameters without altering the surrounding graph.
Self-managed context
Let the model edit its own working file
What it is. Context Language Models treat context as a file the model can edit directly; changes sync immediately into the live context used for the next turn, instead of an external harness deciding what to keep.
Why it exists. Shifting the decision from external rules to the model's own behaviour lets the system learn task-specific strategies for what matters, rather than applying one fixed heuristic everywhere.
Where it breaks. Nobody is now checking the model's own editing decisions the way an external harness would, so a bad edit can compound silently across turns.
How that's handled. On a 24-hour multi-repository agent-swarm task, Context Language Models showed a 65% greater improvement than state-of-the-art context-management strategies at the same compute (arXiv, 2026).
In technical terms
In technical terms: CLMs are built zero-shot from existing models by giving them write access to their own context file, with edits synchronised immediately rather than requeued through a separate management layer.
Compared with what came before
| Full-history prompting | Context-aware orchestration | |
|---|---|---|
| What enters the model | Entire conversation and tool history each call | A budgeted, scored subset per step |
| Who decides what's included | Nobody — everything, by default | A pipeline (or, in CLMs, the model itself) |
| Cost behaviour | Grows with history length regardless of relevance | Reduced 10.2% to 32.2% in ContextRender's benchmarks |
| Failure mode | Dilution: the needed fact is buried, not absent | Omission: the needed fact can still be scored out |
| Auditability | Implicit, hard to reconstruct after the fact | Explicit, if provenance is logged from the start |
The judgement is simple: full-history prompting fails by burying the signal, orchestration fails by occasionally cutting it — but the second failure is visible and measurable, which the first rarely is.
Where it breaks
Context-aware orchestration helps when later steps genuinely depend on specific earlier facts and the pipeline can detect that dependency; it does little for a task that never had an irrelevant-history problem to begin with.
Provenance is not automatic
The Springer Nature paper lists provenance loss, context drift, privacy disclosure and evaluation instability as anticipated failure modes of these systems, not rare exceptions (Discover Artificial Intelligence, Springer Nature, 2026). Logging where a fact came from has to be built in from the start; it cannot be reconstructed cleanly once context has already been trimmed and compacted.
Reuse signals are inferred, not guaranteed
ContextRender's observed-reuse scoring is built from how context was used before (The Latest in AI, 2026). A dependency a later step needs for the first time has no prior reuse history to score against, so it can still be dropped even with dependency tracking in place.
Scheduling layers face the opposite tension: too much deterministic structure around the agents reintroduces the brittleness orchestration was meant to remove, while too little structure loses the auditability enterprises actually asked for. Murakkab's declarative abstraction, which decouples workflow specification from execution configuration via a profile-guided optimizer and adaptive runtime, is one attempt to hold both at once (USENIX OSDI, 2026); its evaluation reports reductions of up to 2.8x GPU usage, 3.7x energy consumption and 4.3x cost compared with state-of-the-art systems while maintaining service-level objectives — a saving that only materialises if the workflow was mis-provisioned to begin with.
Numbers to carry
| Figure | What it measures |
|---|---|
| 10.2%–32.2% | Mean inference cost reduction, ContextRender vs. full history, across AppWorld and 8-objective QA, three execution models, 6K-token budget (2026) |
| 2.8x / 3.7x / 4.3x | GPU usage / energy / cost reduction, Murakkab vs. state-of-the-art serving systems, diverse agentic workflows (2026) |
| 10.3% | Accuracy improvement, Orchestra-o1 over second-best approach, OmniGAIA benchmark (2026) |
| 65% greater improvement | Context Language Models vs. state-of-the-art context management, same compute, 24-hour multi-repository agent-swarm task (2026) |
Myths worth killing
| Commonly repeated | What is actually true |
|---|---|
| "A bigger context window solves the context problem." | A larger window raises the ceiling on what fits; it does not decide what should be sent, which is the actual bottleneck. |
| "Retrieval-augmented generation is the whole solution." | RAG is one acquisition mechanism among several; budgeting, dependency tracking, memory and provenance are separate, necessary layers (Discover Artificial Intelligence, Springer Nature, 2026). |
| "Trimming by recency is good enough." | ContextRender's own evaluation shows recency- and similarity-based truncation can drop information a later step actually depends on (The Latest in AI, 2026). |
| "Letting the model manage its own context is reckless." | Context Language Models showed a markedly larger improvement than state-of-the-art external management on a long multi-repository task, though they trade away the oversight an external harness provides (arXiv, 2026). |
| "Orchestration means losing agentic flexibility." | ORCHESTRA and Orchestra-o1 both combine deterministic control flow with locally adaptive agent behaviour, rather than choosing one over the other (ACL Anthology, 2026; pubdb.com, 2026). |
Worked example
- Papa sets a long-running coding agent swarm to work across several repositories, expecting it to run unattended for roughly a day — illustrative, not a dossier figure.
- Acquisition fetches the relevant files and documentation for the first task; tool calls run a test suite and return results that didn't exist before the step ran.
- Budgeting scores all of this against a fixed token limit — illustratively, a few thousand tokens per step, not the whole swarm's day of history — and only the highest-scoring material reaches the model.
- Dependency tracking flags that a configuration file read six hours earlier is still referenced by a function being written now, and keeps it in context even though it is no longer recent.
- Memory & provenance logs which repository, which commit and which earlier step each surviving fact came from, so Meera, reviewing the output later, can trace a decision back to its source.
- Scheduling routes a sub-task to a specialist code-review agent under the swarm's fixed control flow, while that agent locally adjusts its own retrieval strategy for the specific repository.
- Over the full 24-hour run, a Context Language Model variant of this setup showed a 65% greater improvement than state-of-the-art context-management strategies at the same compute, according to the dossier's reported benchmark on exactly this kind of task.
Should you use it?
Use context-aware orchestration when an agent runs long enough, or calls enough tools, that later steps depend on specific earlier facts and the full history either won't fit or costs too much to resend every time. Skip it for a single-turn task where the whole relevant context already fits comfortably and cheaply in one call — the discipline is overhead there, not protection.
The cheapest first experiment: before adding any dependency graph or memory service, just measure what fraction of each prompt's context the model actually uses versus what gets sent — that gap is the budget you're currently wasting.
Say it in the room
"Don't ask for a bigger context window; ask what decides what the model doesn't see — if nothing does, that's the gap."
Lines to carry
- "More context is not better context."
- "A token budget forces honesty about what matters."
- "Recency is a guess; reuse is evidence."
- "If you can't trace it, you can't trust it."
- "Flexible agents still need a fixed spine."
- "The failure that's visible beats the failure that's buried."
FAQ
Isn't this just prompt engineering with extra steps?
No. Prompt engineering crafts one instruction for one call. Context-aware orchestration governs an ongoing pipeline across many calls, deciding what each step sees, where it came from, and whether it should persist to the next step.
Does retrieval-augmented generation already do this?
Retrieval-augmented generation is one piece — it acquires relevant documents. Orchestration also covers tool outputs, memory, budgeting, dependency tracking and provenance, none of which RAG by itself handles.
If dependency tracking is smarter than recency, why not just use it everywhere?
Because the reuse signal it relies on is inferred from past behaviour. A dependency that's needed for the very first time has no history to score against, so it can still get dropped even with tracking in place.
Is letting the model manage its own context (CLMs) safe?
It performed well on the benchmark reported in the dossier, with a markedly larger improvement than external harness-based management at the same compute, but it removes the external harness that would otherwise catch a bad editing decision before it compounds across turns.
Does more deterministic orchestration mean less agentic flexibility?
Not necessarily. ORCHESTRA and Orchestra-o1 both combine a fixed control flow with agent nodes that adjust prompts and retrieval locally, which is the tension the field is actively working through rather than a settled trade-off.
Where does Murakkab fit into all this?
It sits below the orchestration layer, decoupling how a workflow is specified from how it's executed, so the same declared workflow can be run more cheaply without the people writing it having to think about GPU allocation at all.
Glossary
- Context window — the maximum amount of text a model can consider at once when generating a response.
- Token budgeting — deciding how many tokens to allocate to different pieces of information so the total stays within the model's limit.
- Retrieval-augmented generation (RAG) — fetching relevant external documents and feeding them to a model alongside the query.
- Tool-augmented reasoning — letting a model call external tools and use the results in its reasoning.
- Context drift — a failure mode where fed-in context gradually becomes outdated or inconsistent over a long interaction.
- Provenance — a record of where a piece of context originated, used for auditing and trust.
- Agentic workflow — a sequence of steps carried out by one or more AI agents that can plan, call tools and adapt.
- Service-level objective (SLO) — a target performance or reliability metric a system commits to meeting.
- Multi-agent system — an application composed of multiple cooperating agents, often specialised for sub-tasks.
- Context Language Model (CLM) — a model that manages its own context by treating it as an editable file.
- Observed reuse — a signal indicating how often a past tool result is actually referenced by later steps.
- FLOPs — floating-point operations, a measure of computational work used to compare method efficiency.
- Omnimodal — spanning multiple modalities, such as text, image, audio and video, within one system.
Sources
- Context engineering frames prompt engineering within system level orchestration for large language model applications — Discover Artificial Intelligence, Springer Nature, 2026
- ContextRender: From Execution Dependencies to Agent Context — The Latest in AI, 2026
- Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms — USENIX OSDI, 2026
- Orchestra-o1: Omnimodal Agent Orchestration — pubdb.com, 2026
- ORCHESTRA: AI-Driven Microservices Architecture to Create Personalized Experiences — ACL Anthology, 2026
- Context Language Models — arXiv, 2026