You Can't Improve What You Can't Trace — Agent Observability for Teams

Most teams find out their agent is broken the same way: someone complains. A customer files a ticket, a teammate notices the nightly job produced garbage, a number in a report looks wrong. The failure happened hours or days earlier, somewhere inside a trajectory nobody recorded, and now you are reconstructing a crime scene with no camera footage. This is the default state of agent operations in 2026, and it is a choice — usually an unexamined one. The teams that escape it have figured out that agent observability is its own discipline, not a coat of paint over the application performance monitoring (APM) stack they already run.

The temptation is to treat an agent like any other service: ship it, point Datadog at it, watch the latency and error-rate graphs, page someone when they spike. That works for a request-response API because the unit of work is small, deterministic, and self-describing — a 500 is a 500. It fails for an agent because the unit of work is a trajectory: a multi-step sequence of model calls, tool invocations, and context mutations where every step succeeds with a 200 and the whole thing is still wrong. Your APM dashboard will show you a perfectly healthy agent confidently doing the wrong job. The metrics are green and the outcome is broken, because the metrics were built to watch a different kind of system.

Why agent observability is harder than app observability

Three properties make agents resistant to the tools you already own.

Trajectory reconstruction. A traditional trace is a tree of spans with clean parent-child timing. An agent trace is a narrative — a decision the model made at step 4 that only makes sense in light of what a tool returned at step 2 and what the system prompt said at step 0. To debug it you have to replay the agent’s reasoning, not just its calls. A span saying tool: search → 200 OK, 240ms tells you the search ran; it tells you nothing about whether the agent searched for the right thing, or why it ignored the results. The interesting failures live in the gaps between spans.

Tool-call logging. Every tool call is both an action and a decision, and you need both halves to make sense of it. Logging that the agent deleted a file is APM. Logging that the agent intended a cleanup and then deleted the file is observability. We made this argument at length in Observability for Agents: the meaning is in the correlation between intent and action, and a tool that shows you one without the other gives you no answer, not half an answer. A file write is benign or catastrophic depending entirely on the reasoning that preceded it, and that reasoning is the part your standard logger throws away.

Context-window state. The single most important variable in an agent’s behavior — what was actually in its context window at the moment it decided — is invisible by default and gone the instant the turn ends. The same agent, same code, same task fails differently on Tuesday because Monday’s run left more cruft in the window, or because a retrieved document pushed the relevant instruction out of the model’s effective attention. If you don’t capture the context state at each step, you are debugging a non-deterministic system with the one piece of state that explains the non-determinism deleted. This is the mechanism behind why agents get worse the longer they run, and you cannot see it from a latency graph.

The three signals every agent deployment needs

You do not need fifty dashboards. You need three signals, and most teams track zero of them.

1. Task success rate

Not uptime. Not error rate. The percentage of trajectories that achieved the outcome the user actually wanted. This is the only number that correlates with whether your agent is doing its job, and it is the hardest to capture because success is semantic — it requires a judgment about the result, not a status code. You get it one of three ways: a deterministic check when the task has a verifiable end state (the tests pass, the row was written, the invoice reconciles), a human label on a sample of trajectories, or an LLM-as-judge grading outputs against a rubric — with all the caveats we raised in Who Judges the Judge. Start with the cheapest signal you can compute and widen from there. A team that knows its agent succeeds 71% of the time and can watch that number move is operating in a different universe from a team watching p99 latency and hoping.

2. Trajectory visibility

For any individual run, can you reconstruct what the agent did and why — start to finish — in under five minutes? If the answer is “I’d have to add logging and re-run it,” you have no trajectory visibility, and you will be blind precisely when it matters, because the failures that hurt are the ones you can’t reproduce on demand. This is the capture-the-full-trace discipline: every model call with its prompt and response, every tool call with its arguments and result, and a snapshot of the context state at each step, all stitched into one readable record keyed by a trace ID. The test is concrete — when something goes wrong, you should be opening a trace, not opening your editor.

3. Cost-per-outcome

Not cost-per-token, not cost-per-run — cost per successful outcome. A run that burns 400K tokens and fails costs you more than a run that burns 400K tokens and succeeds, because you will run it again. Dividing total spend by successful outcomes folds your two failure modes — wrong answers and wasteful loops — into one number a non-engineer can read. It also exposes the trap that pure cost dashboards hide: the “cheap” agent that fails half the time and gets retried is more expensive than the “expensive” agent that succeeds first try. Cost-per-outcome is where observability and economics meet, and after the June 2026 billing reset made metered token burn everyone’s problem, it stopped being optional.

The ADLC loop: observability is the front of the feedback cycle

A signal you only watch is a waste of a signal. The point of capturing failure is to stop it from recurring, and that requires a loop — the Agent Development Lifecycle (ADLC) — with observability at the head of it:

1. Observability surfaces a failure mode. Your traces show the agent consistently mishandling a class of input — say, it hallucinates a refund policy when the retrieved doc is ambiguous. You can see it because you captured the trajectory.
2. Eval suites capture the test case. You lift that exact trajectory — the input, the context state, the wrong output — into your eval suite as a regression case. The failure is now a test. This is the discipline we argued for in Your Eval Gate Is Also Code: the eval suite is a living artifact fed by production, not a fixed benchmark you wrote once.
3. Policy updates prevent recurrence. You fix the cause — tighten the prompt, add a guardrail, change the retrieval, constrain the tool — and the eval that used to fail now passes. The fix is verified against the real failure, not your guess about it.

The loop closes: production failures become tests, tests gate fixes, fixes ship, and the next failure feeds the same cycle. Without observability at the front, you have no failures to capture, so your evals drift away from what’s actually breaking and your leaderboard lies about your agent. The trace is what keeps the eval suite honest.

Doing this without a Datadog budget

Here is the part that matters for most of the world’s teams: none of the above requires an enterprise observability contract. The expensive products are convenient, not load-bearing. The discipline is what’s load-bearing, and the discipline is cheap.

• Structured logging is the floor. Emit one JSON object per agent step — trace_id, step, type (model_call / tool_call), the prompt or arguments, the response or result, token counts, and a timestamp. Append it to a file or a single SQLite table. That is a trajectory store. jq over a JSONL file will answer most of the questions a six-figure platform would, and it runs on a laptop.
• Markdown-based state tracking. Have the agent write its plan and running state to a committed markdown file as it works — the same artifact-over-transcript discipline that controls context bloat. When a run fails, the file is a human-readable trajectory of what the agent thought it was doing. You get observability and context hygiene from the same move.
• A simple dashboard is a query, not a product. Three numbers — task success rate, traces-per-day, cost-per-outcome — computed by a nightly script over your JSONL store and printed to a channel. A 30-line Python job posting to Slack beats an unconfigured enterprise dashboard nobody opens. The value was never the rendering; it was deciding which three numbers to compute and looking at them every day.
• Sample, don’t capture everything. You cannot afford to deep-trace 100% of production runs, and you don’t need to. Trace every failure and a random 1–5% of successes. The failures are where the signal is; the success sample is your baseline.

The through-line: observability is a set of decisions about what to record and what to look at, and those decisions are free. The tooling around them ranges from free to expensive, and most teams over-invest in tooling while under-investing in the decisions.

The Southeast Asia angle: observability is budget discipline

For a well-funded team, observability is a reliability investment — fewer incidents, faster debugging, a nicer on-call rotation. For a cost-constrained team, it is something sharper: the only way to spend agent budget responsibly at all. When the token bill is a meaningful fraction of runway — the reality for a four-person team in Phnom Penh, Da Nang, or Cebu, as we argued in the economics of agentic AI — you cannot afford to run agents you can’t measure. Every untraced failed run is money set on fire with no lesson learned.

Cost-per-outcome reframes the whole thing. It turns observability from an ops luxury into the instrument that tells you which agent, which model tier, and which loop design actually earns its tokens. The team that measures it can route the cheap model to the tasks where it succeeds, reserve the frontier model for the tasks that need it, and prove the trade-off with data instead of vibes. That is a capital-light, GPU-free, laptop-portable advantage — exactly the kind of leverage that compounds in markets where every dollar of runway is scarce. You can’t optimize what you don’t measure, and in a cost-constrained market, optimization isn’t a nice-to-have. It’s the game.

The agent you can’t trace is the agent you can’t improve. Start with one JSONL file and three numbers. The rest is iteration.