Skip to content
Telemetry
AI agent monitoring

Monitor AI agent cost, tool use, and outcomes with structured events

Log events when agent runs, tool calls, and model requests finish. Use SQL to compare their cost and reliability with task success and user acceptance.

Connect agent telemetryRead the documentation

Outcomes

  • Measure successful runs, handoffs, retries, and tool failures by workflow and release.
  • Connect token cost and latency with accepted, rejected, or reviewed outcomes.
  • Exclude raw prompts, completions, documents, and tool arguments from events by default.

How it works

How to set it up

1

Record how each run ends

Choose one row per completed agent run or business operation. Record status, duration, workflow, release, handoff, cost, and a safe correlation identifier.

2

Add only useful component events

Log tool and model request events when they answer a known reliability, cost, or quality question. Do not copy entire traces or sensitive payloads.

3

Compare cost with task success

Record whether someone accepted the result, reviewed it, or completed the task. Compare releases with SQL, and check that each rate counts the right runs.

AI agent outcome events in Telemetry

AI agent outcome events in Telemetry

Event tables and the SQL editor used to inspect agent runs, tool calls, and final outcomes.

Boundaries

What this does not replace

  • Telemetry does not ingest OTLP or provide a distributed-trace waterfall. Keep OpenTelemetry traces in an OTLP-compatible backend and correlate selected outcome events with a safe trace or run identifier.
  • Telemetry does not provide LLM trace storage and replay for prompts, completions, retrieved documents, or tool arguments. Send only the fields you have approved for event analysis.
  • Telemetry does not provide built-in prompt management, evaluation datasets, experiments, or scorer execution. Use a dedicated evaluation platform for those tasks.

Try the example

Read the schema, SQL, and sample result

This example includes the schema, read-only SQL, and synthetic results. Use it to check how the query works. It does not measure customer results.

1. Event schema

One row in agent_run_outcomes, with the types used by the query made explicit.

timestamp_utc
Timestamp
workflow
Utf8
model
Utf8
status
Utf8
reviewer_outcome
Utf8
human_handoff
Boolean
Browse event contracts

2. Read-only SQL

Which agent workflows finish successfully and produce accepted outcomes?

SELECT
  workflow,
  model,
  COUNT(*) AS runs,
  100.0 * SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END)
    / NULLIF(COUNT(*), 0) AS technical_success_rate_pct,
  100.0 * SUM(CASE WHEN reviewer_outcome = 'accepted' THEN 1 ELSE 0 END)
    / NULLIF(SUM(CASE
      WHEN reviewer_outcome <> 'not_reviewed' THEN 1 ELSE 0
    END), 0) AS reviewed_acceptance_rate_pct,
  100.0 * SUM(CASE WHEN human_handoff THEN 1 ELSE 0 END)
    / NULLIF(COUNT(*), 0) AS human_handoff_rate_pct,
  AVG(duration_ms) AS average_duration_ms,
  SUM(estimated_cost_usd) / NULLIF(COUNT(*), 0) AS cost_per_run_usd
FROM agent_run_outcomes
WHERE timestamp_utc >= now() - INTERVAL '30 days'
GROUP BY workflow, model
HAVING COUNT(*) >= 20
ORDER BY reviewed_acceptance_rate_pct, runs DESC;

3. Synthetic result

Refund review completes technically but needs substantially more revision or human takeover than the other workflows.

workflowmodelruns
support_triageconfigured-model-a1840
refund_reviewconfigured-model-b620
knowledge_draftconfigured-model-a940
Inspect query, result, and caveats

Capabilities

What is included

Separate event tables for agent runs, model requests, tool calls, evaluations, and product activity
Stable run, operation, trace, account, prompt-version, evaluator-version, and release fields
DataFusion SQL for success rate, p95 duration, cost per accepted result, tool loops, and quality regressions
Dashboards and threshold alerts built from the same reviewed queries
Integration guides for agent frameworks, model providers, OpenTelemetry, and MCP tools

Documentation

Evaluate AI agents with SQLGenAI outcome events with OpenTelemetryOpenAI request-cost implementationOpenAI Agents integration

Setup guides by coding agent

Start with the coding agent already in your workflow

Begin with the agent telemetry skill.md guide, then follow the setup guide for Claude Code, Cursor, or Codex.

Claude Code observability

Ask Claude Code to add event logging while it implements a feature, then verify the events in Telemetry.

Open setup guide

Cursor Telemetry setup

Paste this prompt into Cursor to add event tables, track signup steps, and build dashboards.

Open setup guide

Codex instrumentation prompt

Ask Codex to add product events, verify that they arrive, and report which workflows still need logging.

Open setup guide

See the analysis

SQL recipes that use this capability

Measure AI agent task success and human handoff

Compare agent task success, reviewer acceptance, cost, latency, and human-handoff rate by workflow and model.

Open recipe

Reconstruct a correlated workflow timeline

Reconstruct ordered cross-service workflow steps and elapsed time from a shared workflow identifier.

Open recipe

Detect repeating AI agent tool loops

Find agent runs that repeatedly call a small set of tools without reaching a successful outcome.

Open recipe

Measure accepted AI outputs per dollar

Connect model spend to accepted, saved, or otherwise useful product outcomes.

Open recipe

Calculate LLM cost by feature and model

Attribute model spend, tokens, request volume, and cost per request to product features.

Open recipe

Query nested AI tool-call events

Filter dotted nested fields and rank failing tools without flattening the original event payload.

Open recipe

Customer evidence

How teams use this workflow

"Telemetry is my go-to for dumping logs and tracking events. It's straightforward, the SQL workbench is fast, and I don't have to worry about the schema upfront. Running SQL over a REST API lets me fetch data and dump reports straight to Slack. It's the essentials of what Sentry did for me."

Prem Viswanathan, SwiftCX

Read the customer story

Related capabilities

Related guides and examples

AI-assisted analysis

Describe your question. Get an editable SQL query.

See how it works

Structured event ingestion

Send JSON events. Query them with SQL.

See how it works

SQL workbench and Query API

Query your events in the editor or through the API

See how it works

Start with one production workflow

Use a focused prompt, send synthetic events, and verify the first useful query before expanding coverage.

Open prompt packReview customer evidence