AI Agent Monitoring Beyond the Dashboard

A security review tightens an execution role's permissions on a Tuesday. Nothing pages. Every tool call still returns 200, but the payloads arrive empty, so the agent reports, politely and fluently, that it found nothing to act on. Error rate: flat. Latency: unchanged. Token spend: normal. Resolution quality: degrading quietly until a customer notices days later. Your dashboards saw none of it, because most AI agent monitoring watches the wrong unit.

The scenario is a composite, but only barely. Engineers at AWS documented a near-identical incident inside a production airline booking agent: a missing model-invocation permission produced blank outputs while every infrastructure metric stayed green, and pinning down the root cause took a manual correlation exercise across IAM policies, invocation logs, and orchestration traces, one their team estimated at 30 to 60 minutes even with deep architecture knowledge. Elsewhere in the same system, a poorly scoped supervisor prompt routed roughly 20 percent of requests to the wrong specialist while the metrics stayed flat. The vendor walkthrough is worth reading, but the failures it opens with are the interesting part, because they are structural. Request-scoped observability cannot see them no matter how well it is tuned.

Most LLM agent observability inherits its shape from services that live and die per request. Production AI agents do not. They fail through state drift, contract drift, and silent degradation, three classes of problem that never throw an exception at the layer being watched. This article covers four things: why green dashboards coexist with failing agents, a vendor-neutral failure taxonomy for the loop itself, per-stage instrumentation you can wire into an OpenTelemetry-based stack, and resolution-level SLOs that measure what users actually receive.

The unit of health is the resolution, not the request

A resolution is a goal entering the agent loop and a verified outcome leaving it. "Book the multi-city trip and apply the companion certificate" is one resolution. It spans dozens of requests: model calls, tool invocations, retries, handoffs between specialists, and persistent state that mutates across all of them.

The golden signals chapter that anchors most monitoring practice, with rate, errors, duration, and saturation, was written for request-scoped services. RED-method dashboards ask whether each request succeeded quickly and cheaply. That is a transport question. Token and latency dashboards ask the same question in LLM-specific units. No request knows the goal exists, so no per-request metric can say whether the goal was achieved.

This is why per-request error rate, latency, and token volume can all sit inside normal bounds while resolutions fail: they measure loop mechanics, not outcome quality. An agent can invoke the model flawlessly, call every tool without one transport error, and hand back a fluent answer that fails the user completely. Completion and correctness are separate events, and only one of them raises exceptions.

Two consequences follow. First, watching an agent run and knowing it resolved correctly are different engineering problems with different signals; process signals prove the loop is alive, outcome signals prove it worked. Second, multi-agent systems widen the gap because there is no fixed call graph to baseline. The airline system routes work dynamically between specialists, so a failure can appear at any handoff point and propagate along a path that changes run to run. The stable unit to measure is the resolution, not the path.

A failure taxonomy for AI agent monitoring

The agent failure taxonomy below is not invented from first principles. An empirical study of agent failures audited multi-agent LLM systems and organized what broke into categories spanning system design, inter-agent coordination, and verification. The authors' takeaway, loosely stated, is that coordination and verification deserve as much suspicion as model reasoning. Oriented toward what telemetry can catch, that work collapses into five modes standard dashboards cannot see.

Three mechanisms deserve a closer look, because they explain where the silence comes from.

Permission drift fails politely. When a security team tightens a scope, the tool usually does not start throwing 403s at the agent's layer. It keeps answering 200 with nothing useful inside. The permission change and the symptom live in different systems, days apart. Per-tool emptiness rates close that distance, which is the core of detecting silent failures in LLM agents: measure the payload, not the envelope.

Contract regressions lose data without breaking anything. A renamed field or a newly nullable one flows through as successfully as ever, and the model reasons over a hole. Validating each response against a versioned schema at the tool boundary converts silent data loss into an observable failure at the moment it starts. Consumer-driven contract testing is the established pattern for this class of problem, and it belongs in CI as well as at runtime.

Retry loops are made of healthy requests. Jittered exponential backoff is the correct retry posture, and it still does not eliminate stalls; it only spaces them out. From a per-request dashboard, a wedged loop is a latency chart drifting upward. From a resolution, it is an attempt count climbing toward a budget you set. Those are different observations, and only the second one pages anyone.

What to instrument at each stage of the agent lifecycle

None of this requires adopting a new monitoring vendor. The OpenTelemetry GenAI semantic conventions, including their agent span definitions, are under active development at incubation status, which means agent tracing can extend the collector and backend you already run. The architectural choice that matters comes before any attribute names: anchor the trace context to the resolution, not to the individual request. Model calls, tool calls, retries, and handoffs then become children of one resolution trace, and a single trace ID reconstructs one attempt at one goal.

Dispatch, the start of the resolution

Mint a resolution ID and trace root the moment a goal enters the loop, before the first model call. Record the initial goal in structured form, the caller's identity, the task type for later grouping, and the budgets the loop may spend (maximum steps, wall-clock time, and cost). Without this root, every downstream signal is an orphaned request again.

Tool calls, payload checks beyond status codes

This is where AI agent tool call monitoring earns its keep, and where the status code stops being the signal. For each tool call, log the schema version of the contract in force, whether the response validated against it, and whether the payload was materially empty, alongside the calling principal and the attempt number within this resolution. Emptiness must be a first-class field rather than something inferred later from logs, because permission drift announces itself exactly here. Schema validation at the boundary turns contract regressions into failures you chose to raise, and Pact provider verification formalizes the same check between deploys.

Verification, recording an explicit result

Every resolution should end with a recorded verification event: which verifier ran (a deterministic check, a rule, a human, or a model-based judge) and what it concluded. The verifier's identity matters as much as the result, because a deterministic check on a booking reference and an LLM judge scoring helpfulness are claims of very different strength. If cost forces sampling, sample, but make the rate explicit so verification pass rate stays interpretable. A resolution that completed without verification is a pending claim, not a success.

Handoff, state snapshots and divergence checks

At every handoff between agents, emit a state snapshot: what the agent believes to be true and what a direct query of the system reports. A divergence check compares the two and records the delta. The classic source of divergence is write uncertainty, meaning a call times out after the write committed, then a retry double-applies it. Idempotency keys on every write are the standard mitigation, and Stripe's engineering write-up is still the clearest short case for them. Snapshots are how you notice when something slips through anyway.

Resolution metrics and outcome SLOs

Resolution rate metrics for AI agents answer a different question than request metrics do. Define a small set and hang the SLOs on them:

• Verified resolutions per goal, grouped by task type. This is the agent's real success rate.
• Verification pass rate, the share of completed resolutions whose check actually passed.
• Cost per resolution, summed across every request in the trace, retries included, priced from your real token and tool costs.
• Escalation rate, how often the loop gave up and routed to a human.
• Attempts and steps per resolution, tracked as a distribution, since the tail is where stalls live.

An outcome SLO binds a verified result to a budget: "95 percent of booking goals reach a verified resolution within 15 minutes and 30 steps." The numbers are illustrative, the structure is the point.

Automated quality scoring can carry part of the verification load. AgentCore Evaluations, for example, samples live sessions and scores goal completion with LLM-as-judge methods, and Azure's monitoring guidance pushes the same outcome-oriented direction. Treat those scores as signals rather than ground truth, and calibrate them against human review; even the vendors advise this. The line to hold is simple. Process signals prove the loop ran. Outcome signals prove the goal was met, and the SLO belongs to the second kind.

A five-signal rollout on the stack you already have

Agent instrumentation in priority order, with each step useful on its own:

1. Re-anchor tracing to resolutions. Mint the resolution ID at dispatch and re-parent existing spans. Pure plumbing, no new vendor.
2. Per-tool payload validity and emptiness rates. Catches permission drift and the emptier class of contract regressions within the hour.
3. Per-resolution attempt counters and budgets. Makes stalled loops pageable instead of merely slow.
4. Verification results, sampled first. Publish the pass rate from a sample, then expand coverage as trust builds.
5. Handoff snapshots, for multi-agent systems, with divergence checks at each boundary.

Keep the APM. Transport health, saturation, and provider throttling still matter; they are necessary, just not sufficient. Three pitfalls are worth knowing in advance:

• Metric cardinality. Label by tool and outcome, never by resolution or task ID. The ID belongs in trace and log fields, queryable without exploding your time-series backend.
• Trace sampling. Tail samplers decide per request, which shreds long-lived loops. Sample on resolution outcome instead, and keep every failed resolution at full fidelity.
• Alert fatigue on emptiness. Some tools legitimately return empty results, since a search that finds nothing is correct behavior. Baseline emptiness per tool and alert on deviation from that baseline rather than an absolute threshold.

If you run a framework, check what it already emits before building anything. The OpenAI Agents SDK, for instance, documents its tracing spans for agents, tools, and generations, and those can be bridged into your collector rather than parsed out of logs.

Replay the Tuesday incident under this instrumentation. The emptiness rate on the affected tool spikes within the hour, attempt counters creep as the agent retries hollow responses, and verification pass rate falls as sampling catches unresolved goals. Days of quiet degradation collapse into minutes of signal. That is the difference between watching an agent run and knowing it resolved, and closing it is what AI agent monitoring is for.