olano.ai
Discuss a project

Guide

AI agent observability: what to trace and how, with Langfuse as the worked example

AI agent observability is recording what an agent did on every turn as a trace you can read later: the turn at the top, each model call and tool call nested under it, with timings, tokens, cost, errors and the user's feedback attached, grouped by conversation. Unlike LLM observability, it has to show what the agent did in the world, without copying customers' messages and secrets into another system by default.

Discuss a projectMessage a live agent first
Guide--13 min read

In this guide

What is AI agent observability?

It is being able to answer, after the fact, what an agent did on a turn and why: which model calls and tools it ran, with what result, where the time went and what failed. OpenTelemetry describes observability as understanding a system from the outside without knowing its inner workings, from the traces, metrics and logs it emits; for agents, traces do most of the work.

Agent tracing is the core practice: one trace per turn, built from nested steps (spans). An AI agent observability platform stores those traces, draws each one as a tree, groups them into conversations and lets you filter and score them.

How is agent observability different from LLM observability and APM?

APM watches requests to services, LLM observability watches calls to models, and agent observability watches a loop that mixes both and changes things outside itself. Each keeps the layer below it.

QuestionClassic APMLLM observabilityAgent observability
Unit of workA request or a job.One model call.One turn: many model and tool calls, sometimes a wait for a person.
Typical failureAn error, a timeout, a slow query.A wrong, unsafe or expensive answer.A wrong action, a loop on a failing tool, an unseen approval, a conversation whose cost keeps growing.
SignalsLatency, errors, throughput.Tokens, latency, model and prompt version, quality.All of those, plus tool outcomes, approvals, cost per conversation and user feedback.

The difference is grouping: LLM monitoring shows forty separate model calls; an agent trace shows one turn that made forty because a tool kept failing.

What should you trace for an agent that acts?

Trace each turn as one tree, put timing, cost and outcome on every step, attach approvals and feedback to their turn, and capture content only by choice.

A message arrives: the trace startsModel call: tokens, latency, costTool call: name, duration, outcomeApproval asked and answeredReply sent: the turn closesThumbs up or down: a score on that trace
SignalWhy it mattersHow to capture it
One trace per turnThe turn is what the person experienced.A root span from the message to the final reply (invoke_agent in OpenTelemetry).
Model calls, nestedHow many calls one message caused, and which was slow or costly.A child span per call: model, input and output tokens, latency, finish reason.
Tool calls, nestedWhere the agent changes things.A child span per call (execute_tool): tool name, duration, outcome.
Timing per stepSeparates a slow model, a slow API and a wait for a person.Start and end on every span; an approval wait as its own step.
Cost, rolled upCost per conversation is what a budget review asks for.Cost on each model span, summed per turn and per conversation.
ApprovalsWhat was requested, who decided, how long it waited.An event on the turn; the enforcing system keeps the record.
Errors as levelsFilter failing turns without storing error text.A level or error.type; messages only with content capture.
Conversation groupingMany bugs only make sense across turns.A conversation id on every trace (gen_ai.conversation.id, or a Langfuse session).
The user, pseudonymisedPer-user analysis without copying identities out.A stable pseudonym, never the phone number or email.
Feedback as scoresA thumbs-down should open the turn it judges.A score on that trace, removed if the vote is withdrawn.
Background runsNobody watches a scheduled run live.Trace them too, behind a switch.
Sampling per turnA partial trace hides the tool behind the model call.Keep or drop the whole turn, decided once at the root.
Content off, secrets redactedTraces copy conversations into a second system.Structure and counts by default; content by choice, secrets replaced before export.
Deletion that followsAn erasure request is not met while traces survive.Delete a conversation's traces when the conversation is deleted.

As our prompt injection checklist says, log the action, not the intention.

How much of the conversation should a trace contain?

Start with none of it. Structure (steps, names, timings, tokens, cost and error levels) answers most operational questions and is far less to hand a third party. OpenTelemetry's GenAI conventions agree: the attributes carrying messages, instructions and tool content are Opt-In and flagged as likely to hold personal data. Captured content needs secrets replaced before export, since tool arguments are where keys end up; Langfuse's masking hooks run inside your application. And treat such traces like the conversations themselves: Langfuse can delete a trace, a batch or a whole session.

Where do OpenTelemetry's GenAI semantic conventions fit?

They are the emerging shared vocabulary for agent traces, still marked Development, so treat them as the shape of your data rather than a frozen contract. They define spans for model calls, agents (create_agent, invoke_agent, invoke_workflow) and tools (execute_tool), attributes such as gen_ai.usage.input_tokens and gen_ai.conversation.id, metrics, events and MCP. Since semantic conventions release 1.42 they live in their own repository, with Development status throughout when we checked on 20 October 2026.

OpenTelemetry's post on AI agent observability (March 2025) describes instrumentation built into frameworks or shipped as separate libraries such as Traceloop's OpenLLMetry. Langfuse receives OTLP and maps GenAI attributes to its data model; Datadog says its agent product supports the conventions natively (December 2025). Emit OpenTelemetry-shaped spans to stay portable, and pin the version, because names can still change.

Worked example: how do you turn on Langfuse tracing in Olano?

An admin connects a Langfuse project in Config → Observability (Langfuse), tests it, switches tracing on and chooses what each trace holds. Every agent turn then becomes one Langfuse trace, with each model call and tool call under it, timings, token counts and credits, grouped by conversation. Tracing is off by default and agents cannot change it: AgentFather and an agent's own tools refuse unless an admin is asking. Every field is in the tracing reference.

  1. Create the project on Langfuse Cloud, in the region your traces should live in (EU https://cloud.langfuse.com, US https://us.cloud.langfuse.com, Japan https://jp.cloud.langfuse.com), or on your own server. Nothing is installed on the Olano deployment. Settings → API keys gives a public key (pk-lf-...) and a secret key (sk-lf-...).
  2. Save the keys in Config → Observability with Save keys. They go into the vault, hidden from agents and from the Vault page, and are never shown again. With none saved, the server's LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_BASE_URL are used. Keys alone never turn tracing on.
  3. Set the address and test it. Use the region the keys were created in (Cloud keys work only there) or your server over https://. Test connection checks the keys and remembers the project, which is what lets replies link to their traces.
  4. Choose capture and sampling. Metadata only is the default; Full content shows a warning before you save. Set the share of turns traced (0.0 to 1.0; a turn is traced whole or not at all), an optional environment name such as production, and whether to trace Cortex background runs. Scheduled jobs and conversations are traced either way.
  5. Override one agent if needed. Its Overview tab has an Observability (Langfuse) card where each choice follows Config or is set for that agent, from its next turn with no restart; the guide's example is full content at one turn in five. An admin can give an agent its own keys for a separate Langfuse project.
  6. Read a trace. Admins get a Langfuse link under each traced reply, opening its trace; in Insights, a conversation's Open in Langfuse shows all its traces. Members never see these links.
Capture modeEach trace containsLeft out
Metadata only (default)The structure of the turn, timings, token counts, credits, model and tool names, and errors as a level only.Messages, tool arguments and results, error text. The person's id becomes a stable pseudonym.
Full contentAll of that plus messages, tool arguments and results and error text: what the agent saw and said.Secrets, replaced in both modes.

In both modes every secret the agent could see (vault values, keys in its settings, credential-shaped text) becomes [secret].

Thumbs up and down are kept on the deployment either way, and Insights totals them per agent. On a traced reply the vote is also sent as a user_feedback score on its trace: 1 for up, -1 for down, removed if the vote is taken back. A comment travels only with full content, and votes cast while tracing is off are never sent later. Deleting a conversation deletes its traces in Langfuse, in the background. Traces are sent in the background too, so tracing never slows or breaks a turn: if Langfuse cannot be reached, the turn goes ahead and the settings' Status line shows the problem.

Against the checklist, one gap to know: the user guide does not describe approvals as part of a trace. The record of who approved what is Olano's Approvals inbox and its history.

If something is missing in Langfuse

  • No traces: check the keys, the switch and Test connection; a Cloud refusal almost always means the wrong region.
  • withheld_chars instead of text: Metadata only, working as intended.
  • Missing turns: sampling below 1, or background tracing off.

More in the common problems guide.

Which AI agent observability tools are there?

Alphabetical, not a ranking. Each line checked against the vendor's own documentation or repository on 20 October 2026.

ToolWhat it isWorth knowing
Arize PhoenixArize's open-source AI observability platform for tracing and evaluation, built on OpenTelemetry and OpenInference.Self-hosted; Elastic License 2.0. Arize sells a managed platform, Arize AX.
BraintrustA platform for evaluating and shipping AI products; its SDK covers logging, tracing and evals.Evaluation-first: its quick start is an eval, not a trace.
Datadog LLM ObservabilityTraces each request as spans for every workflow step or agent choice, with cost, latency, quality and safety checks.Now titled Agent Observability in its documentation. A natural fit if you already run Datadog.
HeliconeAn open-source (Apache 2.0) AI gateway and LLM observability platform that logs requests passing through it.Joined Mintlify in March 2026 and runs in maintenance mode.
LangfuseAn open-source AI engineering platform for tracing, evaluation and prompt management, built on OpenTelemetry.MIT core for self-hosting; Cloud has a free Hobby plan and paid plans from US$29 a month. Joined ClickHouse in January 2026. Olano's integration.
LangSmithLangChain's platform to debug, evaluate and monitor LLM applications and agents, with a native LangChain and LangGraph integration.Works with any LLM application and accepts OpenTelemetry traces. Olano does not integrate it.
OpenTelemetry pipelineGenAI instrumentation such as OpenLLMetry or OpenInference, sending OTLP to a collector and any backend that accepts it.Most portable, least opinionated: conversation views and feedback joins are yours to build.

Sources: Arize Phoenix, Braintrust, Datadog, Helicone and its Mintlify announcement, Langfuse, its pricing and ClickHouse announcement, and LangSmith.

Choose by where traces may be stored: if they must stay on infrastructure you run, start with the open-source options. If you write agents on LangChain and want one vendor for tracing, evaluation and deployment, LangSmith fits; outside it, Langfuse's LangChain callback handler is the usual route to LangChain and LangGraph observability. Our Langfuse vs LangSmith comparison goes deeper.

Common mistakes in agent observability

  • Logging full content by default. It copies conversations and pasted credentials into a second system. Start with metadata.
  • Sampling that splits turns. Per-span sampling keeps the model call and drops the tool that caused it. Sample per turn; Langfuse's SDK samples whole traces.
  • Tracing that can fail the request. A synchronous exporter turns a tracing outage into an agent outage. Send in the background.
  • No cost per conversation. Per-call cost hides the conversation that cost twenty times the median. Roll it up, and remember it is a cost line, not a return (see how to measure AI ROI).
  • Feedback not tied to traces. A thumbs-down in a separate table cannot be opened. Score the trace it judges.
  • Keeping traces after the user's data is deleted. Delete by conversation id, in the same operation.
  • Treating traces as evaluation. A trace shows what happened, not whether it was right; that is AI agent evaluation. For LangChain-stack bugs, see debugging LangChain agents.

How Olano helps

Every Olano agent runs on LangChain, LangGraph and Deep Agents (see Deep Agents in production), with the Langfuse integration above built in: metadata by default, secrets replaced, per-agent overrides, feedback scores and deletion that follows the conversation. Around the traces sit the controls that observability only reports on: approvals for outbound and irreversible actions, an append-only audit trail, spending caps and a credential vault resolved per agent. See the Olano platform and security.

The limits: Olano does not host Langfuse (use Langfuse Cloud, at Langfuse's prices, or your own server) and does not integrate LangSmith. The integration sends traces and feedback scores; evaluations and datasets happen in Langfuse.

FAQ

What is observability in an AI agent?

Seeing, after the fact, what an agent did on each turn: every model and tool call in order, with timings, tokens, cost, errors and feedback, grouped by conversation, usually as one trace per turn.

What is an AI agent observability platform?

A backend that stores agent traces, shows each turn as a tree of steps, groups turns into conversations, adds up cost and attaches scores such as user feedback. Langfuse, LangSmith, Arize Phoenix and Datadog offer one.

What is LLM observability?

Monitoring calls to large language models: what each call was sent and returned, its tokens, latency and cost. Agent observability adds the loop around the model: tool calls, approvals, and the turn and conversation they belong to.

What are the top LLM observability tools?

Widely used options include Langfuse, LangSmith, Arize Phoenix, Datadog, Braintrust and Helicone (in maintenance mode since March 2026), plus OpenTelemetry pipelines. Choose by where traces may be stored and whether evaluation or monitoring matters more.

Can Datadog be used for LLM observability?

Yes. Its LLM Observability product, now titled Agent Observability in its documentation, traces model calls, tool calls and agent steps with cost and quality checks, and supports the OpenTelemetry GenAI conventions. It fits best if you already use Datadog.

What are the four pillars of observability?

Commonly metrics, events, logs and traces (MELT). OpenTelemetry's own signals are traces, metrics, logs and baggage, with events and profiles under development or proposed. For agents, traces do most of the work.

Does Olano send my conversations to Langfuse?

Only if an admin turns tracing on; it is off by default and agents cannot change it. Even then the default is metadata only: structure, timings, tokens and credits, with no message text and the person's id replaced by a pseudonym.

Related reading

The guides this one builds on, and the controls behind the traces.

Langfuse vs LangSmith

The two most asked-about tracing platforms.

Debugging LangChain agents

Debug mode, traces, saved state and fake models.

Is your AI agent actually working?

What traces cannot tell you, and how to measure it.

Deep Agents in production

What the harness ships, and what it leaves to you.

Olano security

Isolation, approvals, the audit trail and the vault.

Checked 20 October 2026 against the OpenTelemetry GenAI conventions and OpenTelemetry's documentation sources, Langfuse's documentation, pricing and announcements, the LangSmith, Arize Phoenix, Helicone, Braintrust and OpenLLMetry repositories, Datadog's documentation and blog, and Olano's user guide. Each vendor's own documentation is the primary source. Olano is not affiliated with or endorsed by any vendor named here. If we have described something wrongly, write to support@olano.ai.

Want agents you can see into?

Tell us what your agents do and where their traces may go, and we will tell you how we would trace them.

Discuss a project