What is AI agent observability?

AI agent observability is the practice of capturing every step an autonomous AI agent takes (LLM calls, tool calls, retrievals, sub-agent spawns) as spans in a single trace, so engineers can reconstruct why the agent did what it did. OpenTelemetry formalizes this model in its GenAI semantic conventions, which define dedicated span types such as invoke_agent and execute_tool.

TL;DR

Agent observability extends distributed tracing to non-deterministic workflows: one user request becomes a trace tree of model calls, tool executions, retrievals, and sub-agent invocations.
It is harder than observing a single LLM call because agents loop, branch, cause side-effects through tools, and fail semantically: the trace shows 200 OK while the answer is wrong.
The OpenTelemetry GenAI semantic conventions (status: Development) are the emerging standard, defining span operations like create_agent, invoke_agent, and execute_tool plus attributes such as gen_ai.usage.input_tokens and gen_ai.tool.name.
A useful agent trace records prompts and completions (with PII handling), tool inputs and outputs, token counts, latencies, and the agent's decision path.
Agent telemetry is high-volume, high-cardinality wide-event data; per a 2026 ClickHouse analysis, sampling or rolling it up destroys exactly the context automated debugging depends on.

AI agent observability is the collection and analysis of telemetry from autonomous AI agents, systems where an LLM plans, calls tools, and iterates toward a goal. It records each step as a span in a distributed trace, capturing model inputs and outputs, tool calls, token usage, and latency, so engineers can debug behavior rather than infer it.

The term sits one level above LLM observability, which focuses on individual model calls: prompt, completion, tokens, latency, cost. An agent wraps many such calls in a control loop. Google's agents whitepaper, the definition OpenTelemetry's conventions reference, describes an agent as the combination of a model, tools, and an orchestration layer that reasons and acts in a self-directed fashion. Observing that orchestration layer, not just the model calls inside it, is what makes agent observability its own discipline.

Why is agent observability harder than LLM observability?

Agent observability is harder because agents are non-deterministic programs with loops, branches, and side-effects. A single request can produce dozens of model and tool calls whose count and order differ run to run. Failures are often semantic (the agent confidently returns a wrong answer), so no error status ever fires.

Four properties break the assumptions that single-call LLM monitoring relies on:

Dimension	LLM app observability	AI agent observability
Execution shape	One request → one model call	One request → a variable tree of model, tool, and sub-agent calls
Determinism of structure	Fixed call pattern per code path	Loops and branches chosen by the model at runtime
Failure mode	Timeouts, rate limits, malformed output	Semantic failure: wrong tool, wrong argument, wrong conclusion, with HTTP 200 everywhere
Cost attribution	Tokens per call	Tokens and latency aggregated across every step, sub-agent, and retry in the trace
Side-effects	None beyond the completion	Tool calls mutate real systems: databases, emails, tickets, code

The last two rows drive most production pain. A retry loop that burns 40 extra calls registers only as spend. A tool call that writes bad data succeeds cleanly. Only a full trace of the run, with every decision, input, and output, lets an engineer answer why the agent did what it did.

How do you trace multi-step AI agent workflows?

Multi-step agent workflows are traced with the span-per-step model: the agent invocation is a root span, and every LLM call, tool execution, retrieval, and sub-agent spawn becomes a child span. Standard trace propagation ties them into one tree, and the OpenTelemetry GenAI semantic conventions name each operation and its attributes.

This is the same parent-child span model Google described for microservices in the Dapper paper, applied to reasoning steps instead of RPCs. A support agent that looks up an order, checks a refund policy, and drafts a reply renders as a trace tree like this:

1invoke_agent support-agent                    12.4s
2├── chat gpt-4o (plan next step)               1.1s
3├── execute_tool lookup_order                  0.3s
4├── chat gpt-4o (interpret result)             0.9s
5├── execute_tool get_refund_policy             0.2s
6├── invoke_agent refund-calculator (sub-agent) 4.6s
7│   ├── chat gpt-4o                            1.2s
8│   └── execute_tool compute_refund            0.1s
9└── chat gpt-4o (draft final reply)            2.8s

Each span carries typed attributes from the GenAI conventions. An execute_tool span, for example, looks like this at the attribute level:

1{
2  "name": "execute_tool lookup_order",
3  "attributes": {
4    "gen_ai.operation.name": "execute_tool",
5    "gen_ai.tool.name": "lookup_order",
6    "gen_ai.tool.call.id": "call_a8x2",
7    "gen_ai.tool.call.arguments": "{\"order_id\": \"ORD-4417\"}",
8    "gen_ai.tool.call.result": "{\"status\": \"delivered\", \"total\": 89.90}"
9  }
10}

Model-call spans add gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens, which makes per-request cost attribution a single sum() over the trace. Instrumentation that emits these conventions already exists for the major model providers.

What should you log from an AI agent?

A debuggable agent emits five things per step: the prompts and completions exchanged with the model, the inputs and outputs of every tool call, token counts per call, latency per span, and the decision trace (which action the agent chose and what alternatives it saw). Anything less leaves gaps a post-incident investigation cannot fill.

In practice:

Prompts and completions: the full messages, including system instructions (gen_ai.system_instructions, gen_ai.input.messages, gen_ai.output.messages). These often contain user data, so apply the same PII controls as any log pipeline: redact at the SDK or collector, and treat capture of raw message content as an opt-in decision, which is how the OpenTelemetry conventions classify those attributes.
Tool inputs and outputs: arguments and results for every call. This is where side-effects live, and it is the first place to look when an agent acts on stale or wrong data. Tool-permission policies belong to the adjacent discipline of LLM guardrails.
Token usage and latency: recorded per span, so cost and slowness can be attributed to a specific step. For streaming agents, LLM inference latency metrics such as time to first token belong on the model-call spans.
Decision traces: plan steps, chosen branches, retry counts, and finish reasons (gen_ai.response.finish_reasons), which turn "the agent went weird" into a reproducible sequence.

What does agent telemetry look like at rest?

At rest, agent telemetry is wide events: one row per span with dozens of columns covering trace ID, operation name, model, tool name, token counts, latency, and the full attribute payload. Volumes are large because every user request fans out into many spans, each carrying kilobytes of prompt and tool context.

That shape dictates the storage and query requirements. Debugging a semantic failure means scanning raw spans rather than pre-aggregated dashboards, with questions like "every execute_tool span where lookup_order returned an empty result last month, grouped by agent version":

1SELECT AgentVersion, count() AS failures
2FROM agent_traces
3WHERE SpanAttributes['gen_ai.tool.name'] = 'lookup_order'
4  AND SpanAttributes['gen_ai.tool.call.result'] LIKE '%[]%'
5  AND Timestamp > now() - INTERVAL 30 DAY
6GROUP BY AgentVersion;

High-cardinality fields (trace IDs, tool arguments, user sessions) rule out metrics rollups, and semantic failures are only found in the payloads themselves, so sampling deletes evidence. ClickHouse's April 2026 analysis of retention, sampling, and rollups makes the case that these habits, tolerable for human dashboard workflows, actively break automated reasoning over telemetry: the cited survey data shows at least 90% of practitioners want AI assistance on anomalies and root cause, and over 95% expect those systems to justify conclusions, which requires the raw data to still exist.

Where Langfuse fits for agent observability

Langfuse, the open-source LLM observability platform that ClickHouse acquired in January 2026, is purpose-built for this telemetry. It captures each agent run as a nested trace (model calls, tool invocations, retrieval steps) with token counts and cost per step, and layers on what a general-purpose telemetry store lacks: evaluation pipelines for scoring outputs, prompt management, and a UI organised around inspecting agent runs rather than infrastructure.

The storage properties agent workloads need are the ones described above, and Langfuse gets them from ClickHouse, which it runs on. Columnar compression makes retaining unsampled traces economical, so the trace from a bad agent run three weeks ago is still there at full fidelity, and high-cardinality attributes like tool name or agent version are ordinary columns to filter and group by.

The division of labour with ClickStack, ClickHouse's observability stack for infrastructure and application telemetry, runs the other way: ClickStack is where AI SRE agents do the observing, querying logs, metrics, and traces to investigate production incidents. Observing the agents themselves is Langfuse's job. Langfuse is open source and free to self-host.

FAQ

How do you debug an AI agent that gives wrong answers?

Open the full trace for the failing request and replay the agent's decisions step by step: check whether retrieval returned the right context, whether each tool was called with correct arguments and returned correct results, and where the reasoning diverged. The primary evidence is the raw prompts, completions, and tool payloads from that run.

How do you monitor tool calls made by an AI agent?

Instrument each tool invocation as an execute_tool span carrying gen_ai.tool.name, gen_ai.tool.call.arguments, and gen_ai.tool.call.result per the OpenTelemetry GenAI conventions. Then track per-tool call volume, error rate, latency, and argument patterns over time. Spikes in retries or empty results for one tool usually localize an agent regression to a single integration.

How do you measure AI agent success rates?

Define success per task (resolved without human takeover, correct final answer against an evaluation set, or explicit user confirmation) and record it as an attribute on the root agent span. Success rate is then a grouped aggregation over root spans: by agent version, model, prompt revision, or tool set, which turns prompt changes into measurable experiments.

What are the OpenTelemetry semantic conventions for AI agents?

They are OpenTelemetry's standard vocabulary for GenAI telemetry: span operations including create_agent, invoke_agent, and execute_tool, plus attributes for models, token usage, tool calls, and messages. The conventions have Development status and are maintained in the dedicated OpenTelemetry GenAI semantic conventions repository, with provider-specific pages for Anthropic, AWS Bedrock, Azure AI Inference, and OpenAI.

Is AI agent observability the same as LLM observability?

No. LLM observability looks at individual model calls in isolation; AI agent observability covers the workflow around those calls, tracing planning loops, tool executions, retrievals, and sub-agent invocations as one tree. Agent observability builds on LLM observability the way distributed tracing builds on single-service logging.

What is AI agent observability?

TL;DR