DM Explains · No. 044 · Agents
Traceable agent ecosystems: why agent logs need runs, traces and trajectories
When an agent chains dozens of model calls and tool uses together, a flat log is useless. Here's how run, trace, thread and trajectory turn a tangle of steps into something a human can actually read.
· 14 min read
Read me first
Agent tracing genuinely does turn an unreadable tangle of calls into a debuggable record, but only once someone bothers to flatten it; a raw trace tells you everything and shows you nothing.
A user asks a travel agent to plan a trip over several turns → 1 each flight-search, calendar and weather call is written as a run → 2 runs nest into one trace per operation, bound by a trace ID, capped at 25,000 runs → 3 traces across turns link by thread_id into one session → 4 the nested tree is flattened into an ordered trajectory of messages → 5 an engineer searches “why did it rebook the wrong flight” instead of reading log syntax → 6 a booking subagent's span is captured via execute_task and agent_to_agent_interaction spans, not locked to one vendor → 7 the failure pattern feeds offline evals before the next release
An agent books a flight, checks a calendar, calls three tools and hands off to a subagent, all for one user message — and then gives the wrong answer at turn seven of a long session. Nobody can point at the exact step that went wrong, because nobody wrote down what "the exact step" even means. Without a structured record of every call, tool use and handoff, teams either re-read thousands of raw log lines by hand or shrug and ship the next version hoping it's better.
Everyday picture. Amma writes down every kitchen action the moment it happens — chopping onions, boiling rice, calling Dadi for a tip — like a run. All the lines for tonight's dinner get clipped into one folder, like a trace. Meera links a week of dinner folders together as one ongoing conversation with Papa, like a thread. And Dadi retells the whole week to Bruno as a simple story, first this then that, skipping repeats, like a trajectory.
Where the picture breaks. Dadi's story is sized for a dog's attention span, while a trajectory is built for precise programmatic scoring and search, not casual storytelling, and Amma's diary has no page limit the way a trace is hard-capped at 25,000 runs.
Why it exists now
Agents used to mean one prompt, one answer. Now they mean dozens of chained model calls, tool invocations and subagent handoffs for a single user message, repeated across a multi-turn conversation. A flat application log — timestamp, text, timestamp, text — cannot show which tool call belonged to which reasoning step, let alone which turn of a week-long session caused a downstream failure.
LangChain's observability platform, LangSmith, models this with a vocabulary built for agents specifically: every unit of work is a run, runs for one operation form a trace, traces for one session form a thread (LangChain docs). On 24 September 2026, LangChain pushed this further by launching Trajectories, a chronological, deduplicated view of an entire agent session built on top of that run/trace/thread model, alongside a natural-language "AI Search" over trajectories (LangChain). In parallel, Microsoft's Azure AI Foundry team has been working with Cisco's Outshift to propose new OpenTelemetry spans purpose-built for multi-agent systems (Microsoft), so this is no longer one vendor's private format.
The counterintuitive part: more logging data does not mean more clarity. A complete execution trace of a long agent session can be too detailed to read. The current trend is to deliberately throw structure away — flattening a nested trace into a linear trajectory — to make the agent's behaviour legible again.
How it works
Run capture
Every action gets written down the moment it happens
What it is. Every unit of work an agent performs — an LLM call, a tool invocation, a document retrieval — is recorded as a run, with its inputs, outputs and metadata.
Why it exists. Without an atomic record of each action, there is no raw material to reconstruct what the agent actually did.
Where it breaks. A run on its own shows one step with no sense of which larger operation it belonged to.
How that's handled. Runs are never read in isolation; they are immediately grouped into the next layer up.
In technical terms
In technical terms: in LangSmith, every unit of work an agent performs, such as a model call, tool invocation, or information retrieval, is recorded as a run (LangChain docs).
Trace assembly
Nested runs form one operation's full family tree
What it is. Runs belonging to a single operation are collected into a tree-shaped trace, bound together by a shared trace ID.
Why it exists. Grouping runs preserves the nesting and timing needed to find exactly which sub-call inside a bigger operation failed or ran slowly.
Where it breaks. Very long or highly branched agent operations can run out of room: a trace is hard-capped at 25,000 runs, after which LangSmith rejects additional runs sent for that trace (LangChain docs).
How that's handled. Teams that expect very large fan-out design subagent boundaries to start new traces rather than letting one trace grow without limit.
In technical terms
In technical terms: a trace in LangSmith is a collection of runs for a single operation, bound by a unique trace ID, limited to a maximum of 25,000 runs per trace.
Thread linking
One trace per turn, stitched into a whole session
What it is. A thread is a sequence of traces representing one multi-turn session, grouped by passing a shared thread_id metadata key.
Why it exists. Agents act across many turns, so a single trace of one turn is not enough to see how behaviour evolved across a conversation.
Where it breaks. If a thread_id is never attached to a turn's trace, that turn stays disconnected and the session's arc disappears from view.
How that's handled. Agent frameworks that expect multi-turn use attach thread_id by default, so the discipline sits in the framework rather than relying on every engineer remembering to do it.
In technical terms
In technical terms: a thread in LangSmith is a sequence of traces for a single multi-turn session, grouped by a shared thread_id (LangChain docs).
Trajectory flattening
Deliberately throwing away structure to read it
What it is. A trajectory projects a thread's traces into a flat, deduplicated, ordered list of human, AI and tool messages, with nested run structure stripped out.
Why it exists. A full trace holds too much execution detail to read a session end-to-end; flattening makes the agent's path legible while the underlying trace stays reachable for deeper inspection.
Where it breaks. Flattening loses exact nesting and timing, so a trajectory answers "what happened" far better than "exactly how long did each sub-step take".
How that's handled. Engineers read the trajectory first and drop into the full trace only for the step that needs timing or cost detail.
In technical terms
In technical terms: LangChain launched Trajectories in LangSmith on 24 September 2026 as a chronological, conversational view of an agent session, flattening run nesting while keeping the trace reachable underneath (LangChain).
Search by behaviour
Describe the bug in words, not in log syntax
What it is. AI Search lets an engineer describe a behaviour in plain language and retrieves the trajectories where it occurred, rather than requiring a query language over raw traces.
Why it exists. Because trajectories from different agent stacks share one flat shape, one search method can find a described pattern across all of them rather than needing a bespoke query per framework.
Where it breaks. Search only finds what survived flattening; detail removed when the trajectory was built is invisible to it.
How that's handled. Trajectories work out of the box for traces sent from LangChain, LangGraph, Deep Agents, agent software development kits (SDKs) like OpenAI and Claude, and coding agents like Codex, Claude Code and Cursor, so the same search covers a wide range of agent stacks without custom wiring (LangChain).
In technical terms
In technical terms: LangSmith's AI Search analyses trajectories rather than raw traces to find behavioural patterns described in natural language (LangChain).
Cross-vendor spans
No one vendor should own the record of what agents did
What it is. Microsoft, working with Cisco's Outshift, has proposed new OpenTelemetry spans for multi-agent systems, including execute_task and agent_to_agent_interaction.
Why it exists. Standardised spans let tracing data from different frameworks and vendors be compared with shared tooling, instead of every platform inventing its own incompatible shape.
Where it breaks. This is a proposal in progress built on OpenTelemetry and the W3C Trace Context standard, not yet a universal guarantee that every vendor's traces will line up (Microsoft).
How that's handled. Teams building across multiple agent vendors track the proposal rather than assuming interoperability today.
In technical terms
In technical terms: the proposed OpenTelemetry additions, including agent.state.management and agent_planning attributes, are built on OpenTelemetry and W3C Trace Context to standardise telemetry within multi-agent systems.
Feedback into evals
A trace is only useful once it changes the next version
What it is. Traces and trajectories feed offline evals before release and online evals on live traffic.
Why it exists. Observability only shows what happened; evals turn that record into a judgement that can block or approve the next agent version.
Where it breaks. Without someone acting on what the evals say, the loop never closes and the trace becomes an archive nobody reads.
How that's handled. Offline evals compare a new agent version against known examples before release, while online evals grade live responses where there is typically no pre-written correct answer to compare against (LangChain).
In technical terms
In technical terms: LangSmith's agent development lifecycle connects building an agent to offline evals before release and online evals on live traffic (LangChain).
Illustrative: the trip-planning agent calls flight-search, calendar and weather tools → 1 each call logged as a run → 2 runs bound into one trace, capped at 25,000 runs → 3 traces across turns grouped by thread_id into one thread → 4 nesting removed into a flat trajectory of messages → 5 AI Search queries the trajectory in natural language to spot the booking subagent's failure, feeding the next version
Compared with what came before
| Plain application logging | Traceable agent ecosystem | |
|---|---|---|
| Unit recorded | A request or an error line | A run, with inputs, outputs and metadata |
| Structure | Flat, chronological | Nested trace, grouped into threads |
| Multi-turn view | None by default | Thread links traces across turns |
| Readability of a long session | Requires manual reconstruction | Flattened into a trajectory |
| Search | String or field match | Natural-language AI Search over trajectories (LangChain) |
| Cross-vendor comparison | Ad hoc | Proposed OpenTelemetry spans for multi-agent systems (Microsoft) |
The trade is not free. Plain logging is cheap to write and cheap to store. A traceable agent ecosystem asks every call site to carry a trace ID and, usually, a thread_id, and it asks someone to actually look at the trajectories once they exist. More logging is not more clarity on its own — the clarity comes from the flattening and the search built on top of the capture, not from the capture alone.
Where it breaks
If an organisation stores full conversational and tool-call histories, that record becomes a privacy and retention liability, not just a debugging asset; teams under strict data-retention policy should not assume tracing is free of that cost.
Not a security boundary
A trace records what an agent did, not what it was permitted to do. Capturing tool calls after the fact tells you an agent reached a system it shouldn't have, but it does not stop that call happening. Access control has to sit before the tool invocation, not inside the trace that logs it afterward.
Standard logging and application performance monitoring (APM) record requests and errors, but neither models the agent-specific ideas of nested tool calls, multi-turn threads, or a reasoning path, so bolting generic logging onto an agent and calling it observability misses the structure that makes an agent's failures explicable. OpenTelemetry itself is a generic distributed-tracing standard; the agent-specific layer — runs, threads, trajectories, and now proposed spans like execute_task and agent_to_agent_interaction — is what has to be added on top, and that layer is still being standardised rather than settled.
For a single, stateless model call with no tool use and no multi-turn context, this entire apparatus is overkill; simple prompt-response logging is enough, and reaching for runs, traces and threads there is complexity without payoff.
Numbers to carry
| Figure | Value |
|---|---|
| Maximum runs per trace in LangSmith | 25,000, after which additional runs are rejected (LangChain docs) |
| Trajectories launch date | 24 September 2026 (LangChain) |
| Frameworks Trajectories work with out of the box | LangChain, LangGraph, Deep Agents, OpenAI and Claude SDKs, and coding agents including Codex, Claude Code and Cursor (LangChain) |
| Proposed OpenTelemetry multi-agent spans | execute_task, agent_to_agent_interaction, among others (Microsoft) |
| LangSmith SDK languages | Python, TypeScript, Go, Java (LangChain) |
Myths worth killing
| Commonly repeated | What is actually true |
|---|---|
| "If you log everything, debugging is automatic" | A full trace can be too detailed to read; trajectories exist precisely because raw capture isn't legible on its own |
| "Tracing is just OpenTelemetry" | OpenTelemetry is a generic standard; agent-specific concepts like runs, threads and trajectories are built on top of it, and multi-agent spans are still a proposal in progress |
| "Observability fixes the agent" | A trace only matters once it feeds an eval that changes the next version; watching is not improving |
| "One trace is one conversation" | One trace is one turn; a thread is the conversation, built by linking traces with a shared thread_id |
| "Tracing is a security control" | Tracing records what happened after the call; it does not stop an unauthorised tool call before it runs |
Worked example
- Amma — in the running example, the user — asks the trip-planning agent to plan a holiday; this single message starts turn one.
- The agent — records a run for its first model call, then a run for a flight-search tool invocation, then a run for a calendar check; these three runs nest under one trace for turn one.
- The agent — hands part of the task to a booking subagent, which records its own nested runs inside that same trace, since it belongs to the same operation.
- The system — tags the trace with a shared thread_id so it links to the traces of turns two through seven as the user keeps revising the plan.
- At turn seven — the agent returns the wrong departure date; an engineer opens the thread and reads its trajectory, the flattened, deduplicated message list, rather than the raw nested trace.
- The trajectory — shows, in order, that the calendar tool returned a date in the wrong timezone back at turn three, and every later turn repeated it without re-checking.
- Illustrative: if that trace had instead branched into, say, 30,000 nested runs across hundreds of tool retries, it would have exceeded LangSmith's 25,000-run cap and the later runs would have been rejected outright, hiding the failure rather than merely burying it.
- The team — writes this case into an offline eval so the next agent version is tested against exactly this timezone scenario before release.
Should you use it?
Reach for structured run/trace/thread/trajectory tracing when an agent chains multiple model calls, tool invocations or subagent handoffs per request, when the agent runs across multiple turns, or when engineers need to search across large volumes of sessions for a described behaviour rather than reading each one by hand. Skip it for a single stateless call with no tools, and think twice about storing full histories at all where data-retention policy forbids it.
The cheapest first experiment is to add trace IDs and a thread_id to one existing multi-turn agent, look at a week of its trajectories, and see whether the failure that's currently a mystery turns into a two-line explanation.
Say it in the room
"Trace every run, link every turn into a thread, then flatten it into a trajectory before anyone tries to read it — the nesting is for the machine, not for the human doing the debugging."
Lines to carry
- "A run is one step; a trace is the tree; a thread is the conversation."
- "More logging is not more clarity."
- "Flattening on purpose is how you make a tangle readable again."
- "A trace only matters once it changes the next version."
- "Tracing is not a security boundary; access control happens before the call, not in the log of it."
- "25,000 runs per trace is a ceiling, not a guarantee."
FAQ
Is a trace the same thing as a trajectory?
No. A trace is the full nested tree of runs for one operation, kept for debugging timing and cost. A trajectory is a flattened, deduplicated projection of a whole thread's traces into one readable list of messages, built specifically so a human or a search tool can follow a session without wading through nesting.
Do I need a trace ID on every single model call?
Every run needs to belong to a trace to be useful at all; without a shared trace ID, runs sit as disconnected fragments with no sense of which larger operation they were part of.
What happens if an agent's operation produces more than 25,000 runs?
In LangSmith, that trace hits its hard cap and further runs sent for it are rejected, so extremely long-running or heavily branched operations need to be split across multiple traces rather than left to grow without limit.
Does standardised tracing across vendors already exist?
Not fully. Microsoft and Cisco's Outshift have proposed new OpenTelemetry spans for multi-agent systems, built on OpenTelemetry and W3C Trace Context, but this is a proposal in progress rather than a settled guarantee that every agent framework's traces will already line up.
Does tracing replace evals?
No. Traces and trajectories are the record of what happened; offline and online evals are what judge that record and feed improvements back into the next agent version. A trace that nobody evaluates just sits there.
Glossary
- Agent — a software system that uses an AI model to plan and take multi-step actions, such as calling tools or other agents, to complete a task.
- Run — a single recorded unit of work an agent performs, like one model call or one tool use.
- Trace — the full nested tree of runs that make up one operation, linked by a shared trace ID.
- Thread — a sequence of traces representing one multi-turn conversation or session.
- Trajectory — a flattened, ordered, deduplicated list of messages showing the path an agent took across a session, with nested execution detail removed.
- Span — in OpenTelemetry terms, a single timed unit of work within a trace, roughly equivalent to a run.
- OpenTelemetry — an open standard and set of tools for collecting telemetry data (traces, metrics, logs) in a vendor-neutral format.
- Offline eval — a test that compares a new agent version's output against known, pre-written example answers before release.
- Online eval — a method of scoring an agent's live production responses after release, typically without a pre-written correct answer.
- Multi-agent system — an AI system composed of multiple cooperating agents, often including subagents handling specialised parts of a task.
- Tool invocation — an instance of an agent calling an external function, API or service to retrieve information or perform an action.
- Observability platform — a tool that collects and visualises runtime data from a system so engineers can understand and debug its behaviour.
- Semantic convention — a standardised naming and structuring scheme for telemetry data so different tools can interpret it consistently.
Sources
- Observability concepts - Docs by LangChain — LangChain
- Trajectories now in LangSmith: A readable view of every agent session — LangChain, 2026
- Search agent traces by behavior with AI Search in LangSmith — LangChain, 2026
- AgentGraph: Trace-to-Graph Platform for Interactive Analysis and Robustness Testing in Agentic AI Systems — AAAI
- Azure AI Foundry: Advancing OpenTelemetry and delivering unified multi-agent observability — Microsoft
- What is LangSmith? — LangChain
- LangSmith: Agent & LLM Observability Platform — LangChain