DM Explains · No. 026 · Architecture
Context rot: why your AI agent can be confident, well-formed and still wrong
An agent can produce a correct-looking answer from context that is stale, partial or scoped to the wrong task — and neither the model nor the pipeline will report an error.
· 15 min read
Read me first
Context rot genuinely does make agents fail in production, but the failure never looks like a failure — the model answers fluently, the pipeline reports healthy, and the only thing wrong is the context nobody checked.
The multi-step support agent: 1 fans out to history, billing and KB → 2 each hop gets a named span → 3 the KB hit fails provenance → 4 a stale-context metric is emitted → 5 the run scores low though the model sounds confident → 6 the exact failing run is reproduced and fixed, not guessed at
Amma keeps a running shopping list stuck to the fridge with a magnet. Through the week, Dadi crosses off what she's used, Papa adds items the electrician mentioned, Arjun scribbles in a snack he wants. Nobody tells anyone else. By Saturday the list looks complete and tidy — neat handwriting, no gaps, nothing crossed out twice — except half of it is last week's menu, and there is no way to tell which half just by looking at it.
Production AI agents now chain retrieval, tool calls, memory lookups and sub-agent handoffs into a single customer request. A stale or wrongly scoped piece of context introduced at any one of those steps propagates silently through the rest, and the system keeps reporting green all the way to a wrong refund, a wrong account update, a wrong answer sent to a customer who trusted it.
Everyday picture. Amma's fridge shopping list looks complete and tidy — but Dadi, Papa and Arjun have each silently added and crossed off items all week without telling each other. The version on the fridge is confidently wrong. It just looks fine from across the kitchen.
Where the picture breaks. A fridge list has no built-in timestamp or source tag for each item. Real context observability explicitly tracks freshness and provenance per item, something paper and a magnet cannot do on their own.
Why it exists now
Through 2026, cloud and observability vendors moved from generic "log the LLM call" tooling to agent-specific telemetry, because generic logging could not answer the question that actually mattered: which step, in a chain of ten, handed the model bad context? AWS launched CloudWatch generative AI observability (Preview) for end-to-end tracing across language models, agents, knowledge bases and tools (Launching Amazon CloudWatch generative AI observability — AWS), then followed with CloudWatch Omni, which adds eval-driven observability scoring correctness, groundedness and tool selection on every single invocation (CloudWatch Omni — AWS). Amazon Bedrock AgentCore Observability separately added capture of token usage, tool-selection patterns and reasoning traces across agent workflows (Build trustworthy AI agents with Amazon Bedrock AgentCore Observability — AWS).
In parallel, OpenTelemetry published GenAI Semantic Conventions to standardise how agent operations get written down as structured spans (AI Agent Monitoring: Why Traditional APM Falls Short — LangChain). That standard matters because a Partnership on AI assessment of agent frameworks found that most of the frameworks it reviewed do not yet emit the deterministic fields needed to detect these failures at all (The Observability Gap in AI Agents — Partnership on AI). Put together: the tooling exists now because teams discovered, the expensive way, that a healthy model and a healthy pipeline say nothing about whether the context between them was any good.
How it works
Fan-out
One request, many places context can rot
What it is. A single agent request explodes into a chain: an inference call, several parallel tool invocations, a memory lookup, maybe a handoff to a sub-agent.
Why it exists. Agents are built this way because no single model call can retrieve an account history, check a billing system and issue a refund in one shot. Each stage does one job and passes its output to the next.
Where it breaks. Each of those stages is an independent chance to read stale context or write a wrongly scoped result forward. Infrastructure metrics like function duration or API gateway latency give no signal about which step actually caused a slow, wrong or expensive response (AWS Well-Architected Agentic AI Lens — AWS).
How that's handled. Treat the chain as a sequence of named, inspectable operations, not one opaque box.
In technical terms
In technical terms: the AWS Well-Architected agentic-ai lens recommends agent-aware telemetry that decomposes execution into observable, attributable operations rather than relying solely on infrastructure-level metrics.
Span logging
Name every step so you can blame the right one
What it is. Every operation — one inference call, one tool call, one retrieval — gets recorded as a structured span with a name, timing and attributes.
Why it exists. Without a name attached to each step, a failure can be seen but not located; "the agent was wrong" tells you nothing about where in six steps it went wrong.
Where it breaks. Logging only works if the framework actually emits these fields in the first place. The Partnership on AI's review found most of the four agent frameworks it assessed do not yet emit the deterministic fields these spans need, so the logging exists on paper but is incomplete in practice (The Observability Gap in AI Agents — Partnership on AI).
How that's handled. Standardise on a shared span format before building dashboards on top of it, rather than inventing a bespoke schema per team.
In technical terms
In technical terms: OpenTelemetry's GenAI Semantic Conventions standardise how inference, tool calls and retrieval are written as spans, covering aspects traditional OpenTelemetry conventions did not anticipate (AI Agent Monitoring — LangChain).
4 checks
Freshness, completeness, provenance, scope
What it is. Context observability scores the context an agent actually used, at the moment it acted, on four separate dimensions: is it current, is it whole, can it be traced to a source, and is it the right context for this task.
Why it exists. This sits in the gap between data observability, which watches pipelines and tables, and model observability, which watches calls and tokens — the exact place where a healthy pipeline can still hand a stale or wrongly scoped document to a healthy model (Context Observability for AI Agents — Atlan, 2026).
Where it breaks. These four checks need some baseline telemetry already in place to run against; they cannot be bolted onto a data source that was never instrumented at all.
How that's handled. Instrument the source before trying to score what came out of it.
In technical terms
In technical terms: context observability is positioned explicitly between data observability (pipelines, tables) and LLM observability (model calls, tokens), scoring freshness, completeness, provenance and scope-correctness as four distinct dimensions.
Emit metrics
Nothing useful is captured for free
What it is. Derived signals — task success rate, cache hit rate, tool-selection accuracy, time-to-first-token, cost per task — do not appear automatically; they must be deliberately emitted as custom metrics, tagged by agent ID, workflow, environment, task type and model ID.
Why it exists. A dashboard can only show what something upstream decided to measure and label.
Where it breaks. If a framework never exposes the underlying deterministic fields, no amount of dashboard engineering can produce these metrics after the fact — this is the same gap the Partnership on AI flagged in most of the frameworks it assessed (The Observability Gap in AI Agents — Partnership on AI).
How that's handled. Audit what a framework actually emits before choosing it for anything customer-facing.
In technical terms
In technical terms: metrics such as task success/failure rate, cache hit rate, tool-selection accuracy, time-to-first-token and cost per task are custom-emitted from the agent or OpenTelemetry collector, dimensioned by agent ID, workflow, environment, task type and model ID (AWS Well-Architected Agentic AI Lens — AWS).
Score live
Grade every run, not just the ones that complain
What it is. Correctness, groundedness and tool selection get scored continuously, on live production traffic, not only when a user files a complaint.
Why it exists. Confident is not the same as current — an agent can produce a fluent, well-formed answer from rotten context, and nothing short of continuous scoring will catch it before a customer does.
Where it breaks. This finds regressions after they have already shipped to production; it does not stop the first bad piece of context from entering the chain in the first place.
How that's handled. Pair online evaluation against live traffic with offline evaluation against versioned edge-case datasets, so both drift and known hard cases get checked.
In technical terms
In technical terms: CloudWatch Omni provides eval-driven observability giving continuous quality scores on every agent invocation, running online evaluations against production traffic and offline evaluations against versioned datasets built from edge cases (CloudWatch Omni — AWS).
Close loop
Reproduce the failing run, don't guess
What it is. A flagged failure gets opened against its own exact trace, a different prompt or configuration is tried against that real failing data, scored with evaluators, compared to a dataset built from production traffic, and the better variant is shipped while evaluators keep watching.
Why it exists. Fixing a context-rot failure by guessing at a general-purpose patch tends to fix the symptom in the demo and miss the actual run that failed.
Where it breaks. This closes the gap between detecting and fixing a known failure pattern. It will not catch a genuinely new kind of scope error the evaluators were never built to check for.
How that's handled. Keep adding newly discovered failure patterns back into the evaluator set, so the loop widens over time rather than staying fixed.
In technical terms
In technical terms: this close-the-loop workflow runs end to end: instrument and test an agent, deploy and monitor it, catch a quality regression on live traffic, reproduce and fix it against the exact failing runs, then set up continuous evaluation to catch recurrence (Introducing Amazon CloudWatch Omni — AWS).
Illustrative: refund ticket #4821 → 1 trace emits 1,200 tokens, 3 tool calls → 2 scored: provenance low, scope mismatch → 3 flagged rotten (stale KB snippet) → 4 replay shows refund sub-agent read a cached policy from last quarter → 5 pipeline patched to invalidate cache on policy version bump → 6 offline suite re-run on 40 edge cases, all pass → shipped
Detecting rot in production and closing the loop to fix it are different jobs with different tools: one scores every run as it happens, the other reproduces one specific failing run on demand. Treating them as the same step is why many teams can see that something is wrong for weeks before anyone fixes the actual cause.
Compared with what came before
| Infrastructure monitoring (old way) | Context + agent observability (new way) | What you give up | |
|---|---|---|---|
| What it watches | Latency, error codes, CPU, API responses | Spans per step, context freshness/completeness/provenance/scope, live eval scores | Simplicity — more to instrument, more to maintain |
| Can it see a wrong-but-fluent answer | No | Yes, via continuous correctness/groundedness scoring | Nothing fixed; it's additive |
| Attributes blame to a step | No, only "the request was slow or errored" | Yes, via named spans | Needs the framework to actually emit the fields |
| Catches a stale document that fits the context window fine | No | Yes, if freshness is checked | Requires instrumenting the source, not just the model |
The judgement is plain: infrastructure monitoring tells you the house didn't catch fire, nothing more. It was never built to tell you the shopping list on the fridge is half a week out of date, and no amount of staring harder at latency graphs will make it do that job.
Where it breaks
Context and agent observability earns its keep when an agent chains multiple steps that each touch context — retrieval, tool calls, memory, sub-agents. It stops earning its keep, or simply cannot be built, in a few situations.
Needs a framework that actually emits the fields
A Partnership on AI assessment of agent frameworks found that most of the frameworks it assessed do not yet emit the deterministic fields these monitoring signals depend on. Choosing a framework without checking this means the dashboards promised by vendors simply have nothing to draw from.
Not a fix for an uninstrumented source
Context observability needs some baseline telemetry already in place to build on. It cannot retroactively check the freshness or provenance of a data source that was never instrumented at all — there is nothing to check against.
Live scoring finds regressions, not root causes
Continuous evaluation against production traffic will flag that quality dropped. It does not, on its own, tell you a stale document was the cause; that still requires opening the specific failing trace and reproducing it.
A simple single-call system with no retrieval, memory or tool use has very little context to rot, and bolting this machinery onto it is overkill. And none of this replaces standard application performance monitoring for pure infrastructure failures, such as a database connection pool running out — that is still a plumbing problem, not a context one.
Numbers to carry
| Figure | What it means |
|---|---|
| Over 40% of agentic AI projects projected cancelled by 2027 (Gartner, as cited 2026) | A warning about cost, value and risk-control failures in agent deployments broadly, not specifically about context rot |
| 4 dimensions | The number of separate things context observability checks: freshness, completeness, provenance, scope-correctness |
| 4 frameworks assessed | How many agent frameworks the Partnership on AI reviewed; most did not emit the deterministic fields needed |
Myths worth killing
| Commonly repeated | What is actually true |
|---|---|
| "If the model isn't hallucinating, the answer is reliable" | An agent can produce an answer well-grounded in the context it was given, and still be wrong, because that context was stale, partial or scoped to the wrong task |
| "A bigger context window fixes this" | Context window size is about how much text fits in one call; context rot is about whether what's placed in that window is current, complete, traceable and correctly scoped — size doesn't touch any of that |
| "If the pipeline dashboard is green, the agent is fine" | Infrastructure monitoring like latency and error rates cannot see context quality at all; it only sees whether services responded |
| "Logging the LLM calls is the same as observing the agent" | Most agent frameworks assessed by the Partnership on AI do not yet emit the deterministic fields needed to derive the signals that actually catch context rot |
Worked example
- Customer messages the support agent asking why a refund hasn't arrived.
- Agent retrieves account history from a knowledge base, calls a billing tool for transaction status, and hands off to a refund sub-agent — three steps, each touching context independently.
- Billing tool returns a transaction record that was cached before a status change went through; this is stale context, invisible to the model, which simply predicts text from whatever record it was given.
- Refund sub-agent confidently tells the customer the refund is still processing, grounded entirely in the stale record it received — no hallucination occurred, the context itself was wrong.
- Span logs, if the framework emits them, show exactly which step (the billing tool call) handed over the stale record, rather than leaving the team to guess across the whole chain.
- Live scoring flags the interaction as a correctness or groundedness failure on production traffic, days before any customer complaint reaches a human.
- Engineer opens the exact failing trace, confirms the cache was serving an outdated record (illustrative: say the cache was roughly six hours stale), fixes the cache invalidation rule, and reruns the evaluator against the same trace to confirm it now passes.
Should you use it?
Build this when agents in production chain several steps that each touch context, when a failure needs to be traced back to a specific cause rather than just rerun and hoped away, and when the volume of agent traffic makes manual review impossible. Skip it for a single-call prompt-response system with no retrieval or tool use, and don't expect it to substitute for standard infrastructure monitoring on plumbing failures that have nothing to do with context.
The cheapest first experiment is not a new dashboard — it's an audit. Pick the framework already in use and check, directly, which deterministic fields it actually emits per step before buying or building anything on top of it.
Say it in the room
"Use it when agents chain multiple context-touching steps in production and failures need tracing to a cause; skip it for single-call systems with nothing to decay."
Lines to carry
- "Confident is not the same as current."
- "The model can be fine and the context still rotten."
- "A healthy pipeline can still hand over a stale answer."
- "Nothing is observed unless something emitted it."
- "Fetching something is not the same as fetching the right thing."
- "Detecting rot and fixing it are two different jobs."
FAQ
Is context rot the same as the model hallucinating?
No. Hallucination is the model inventing something not supported by its context. Context rot is the context itself being stale, incomplete, untraceable or scoped wrong — the model can be entirely faithful to what it was given and still be wrong, because what it was given was bad.
Will a bigger context window stop this?
No. Context window size is a limit on how much text fits into one call. Context rot is about the quality of what gets placed inside that window, which can go wrong at any size, including a window with plenty of room to spare.
We already log our LLM calls — isn't that enough?
Usually not, by itself. A Partnership on AI review found most agent frameworks it assessed don't yet emit the deterministic fields that the deeper signals — tool-selection accuracy, context freshness, groundedness — actually need. Logging that a call happened is different from logging enough to diagnose why it was wrong.
Where does this fit next to normal application monitoring?
Alongside it, not instead of it. Standard infrastructure monitoring still matters for plumbing failures like a database running out of connections. Context and agent observability covers the layer infrastructure monitoring was never built to see: whether the content handed to the model was any good.
How do we know this is working rather than just adding dashboards?
Check that a flagged quality regression can actually be traced back to one named step, reproduced against its exact failing run, and shown to improve after a fix — not just that a chart exists. If nobody can point to a span and say "this one," the instrumentation isn't finished yet.
Glossary
- Context rot — the gradual degradation of an AI agent's effective context quality, through staleness, missing pieces, loss of traceability, or wrong scope, even when the model itself works correctly.
- Context window — the maximum amount of text a model can process in a single call; a size limit, not a quality measure.
- Telemetry — data automatically emitted by a running system, used to observe its behaviour.
- Span — a single recorded step in an agent's execution, such as one tool call, that can be timed and inspected.
- Trace — the full sequence of spans making up one end-to-end agent request.
- OpenTelemetry (OTel) — an open standard and toolset for collecting and formatting telemetry such as traces, metrics and logs.
- GenAI Semantic Conventions — OpenTelemetry's standardised field names and structures for recording generative AI and agent operations.
- Eval-driven observability — monitoring that continuously scores output quality, rather than just tracking whether a system is running.
- Groundedness — a measure of whether a model's answer is actually supported by the source material it was given.
- Provenance — the ability to trace a piece of context back to where it originally came from.
- Scope-correctness — whether the context used was the right context for this particular agent and task, as opposed to context meant for something else.
- Context drift — context that was accurate when captured becoming stale or misaligned as the underlying data changes over time.
- APM (application performance monitoring) — traditional tooling tracking infrastructure health such as latency, error rates and resource usage.
- AgentCore — Amazon Bedrock's managed runtime and tooling for building and running AI agents, including its Observability feature.
- Tool selection accuracy — a measure of whether an agent correctly chose which external tool or API to call for a given step.
The fridge list was never wrong because someone lied on it. It was wrong because nobody tagged who wrote what, or when. Context rot in agents is the same quiet failure at industrial scale — the fix was never to trust the model harder, but to make every piece of context say where it came from and how fresh it still is.
Sources
- Monitor AI agents - Amazon CloudWatch — AWS, n.d.
- Tutorial: Fix a production quality regression - Amazon CloudWatch — AWS, n.d.
- AWS Well-Architected Agentic AI Lens — Agent performance best practice — AWS, n.d.
- CloudWatch Omni — AWS, n.d.
- Context Observability for AI Agents: The 4 Dimensions — Atlan, 2026
- AI Agent Monitoring: Why Traditional APM Falls Short — LangChain, n.d.
- The Observability Gap in AI Agents: What to Fix Before Agents Scale — Partnership on AI, 2026
- Build trustworthy AI agents with Amazon Bedrock AgentCore Observability — AWS, n.d.
- Introducing Amazon CloudWatch Omni: Observability for the AI Era — AWS, n.d.
- Launching Amazon CloudWatch generative AI observability (Preview) — AWS, n.d.