1. Event contract
One row in agent_run_outcomes, with the types used by the query made explicit.
- timestamp_utc
- Timestamp
- workflow
- Utf8
- model
- Utf8
- status
- Utf8
- reviewer_outcome
- Utf8
- human_handoff
- Boolean
How it works
A reviewable workflow from signal to decision
1
Define the terminal outcome
Choose one row per completed agent run or business operation. Record status, duration, workflow, release, handoff, cost, and a safe correlation identifier.
2
Add only useful component events
Log tool and model request events when they answer a known reliability, cost, or quality question. Do not copy entire traces or sensitive payloads.
3
Join operation with value
Record acceptance, human review, task success, or another explicit product outcome, then compare releases with reviewed SQL and complete denominators.
AI agent outcome events in Telemetry
An actual Telemetry workspace showing the event tables and SQL surface used to inspect agent runs, tool calls, and terminal outcomes.
Boundaries
What this does not replace
- Telemetry does not ingest OTLP or provide a distributed-trace waterfall. Keep OpenTelemetry traces in an OTLP-compatible backend and correlate selected outcome events with a safe trace or run identifier.
- Telemetry does not store and replay prompts, completions, retrieved documents, or tool arguments as a dedicated LLM trace product. Send only approved, bounded fields.
- Telemetry does not provide built-in prompt management, evaluation datasets, experiments, or scorer execution. Retain a specialist evaluation platform when those workflows are required.
Inspectable proof path
From event contract to a visible answer
This example uses a declared schema, read-only SQL, and deterministic synthetic results. It demonstrates the workflow without presenting sample data as a customer benchmark.
2. Read-only SQL
Which agent workflows finish successfully and produce accepted outcomes?
SELECT
workflow,
model,
COUNT(*) AS runs,
100.0 * SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END)
/ NULLIF(COUNT(*), 0) AS technical_success_rate_pct,
100.0 * SUM(CASE WHEN reviewer_outcome = 'accepted' THEN 1 ELSE 0 END)
/ NULLIF(SUM(CASE
WHEN reviewer_outcome <> 'not_reviewed' THEN 1 ELSE 0
END), 0) AS reviewed_acceptance_rate_pct,
100.0 * SUM(CASE WHEN human_handoff THEN 1 ELSE 0 END)
/ NULLIF(COUNT(*), 0) AS human_handoff_rate_pct,
AVG(duration_ms) AS average_duration_ms,
SUM(estimated_cost_usd) / NULLIF(COUNT(*), 0) AS cost_per_run_usd
FROM agent_run_outcomes
WHERE timestamp_utc >= now() - INTERVAL '30 days'
GROUP BY workflow, model
HAVING COUNT(*) >= 20
ORDER BY reviewed_acceptance_rate_pct, runs DESC;3. Synthetic result
Refund review completes technically but needs substantially more revision or human takeover than the other workflows.
| workflow | model | runs |
|---|---|---|
| support_triage | configured-model-a | 1840 |
| refund_review | configured-model-b | 620 |
| knowledge_draft | configured-model-a | 940 |
Capabilities
What is included
Named event tables for agent runs, model requests, tool calls, evaluations, and product outcomes
Stable run, operation, trace, account, prompt-version, evaluator-version, and release fields
DataFusion SQL for success rate, p95 duration, cost per accepted result, tool loops, and quality regressions
Dashboards and threshold alerts built from the same reviewed queries
Integration guides for agent frameworks, model providers, OpenTelemetry, and MCP tools
Agent-specific setup paths
Start with the coding agent already in your workflow
Each guide carries the same reviewed telemetry boundaries into a focused Claude Code, Cursor, or Codex implementation path.
Claude Code Observability
Give Claude Code a prompt that makes telemetry part of the implementation pass instead of a separate cleanup project.
Open setup guideCursor Telemetry Setup
A copyable setup prompt for adding event tables, funnel tracking, and dashboards from inside Cursor.
Open setup guideCodex Instrumentation Prompt
A focused prompt that asks Codex to instrument the rest of the product, verify events, and summarize coverage gaps.
Open setup guideSee the analysis
SQL recipes that use this capability
Measure AI Agent Task Success and Human Handoff
Compare agent task success, reviewer acceptance, cost, latency, and human-handoff rate by workflow and model.
Open recipeReconstruct a Correlated Workflow Timeline
Reconstruct ordered cross-service workflow steps and elapsed time from a shared workflow identifier.
Open recipeDetect Repeating AI Agent Tool Loops
Find agent runs that repeatedly call a small set of tools without reaching a successful outcome.
Open recipeMeasure Accepted AI Outputs per Dollar
Connect model spend to accepted, saved, or otherwise useful product outcomes.
Open recipeCalculate LLM Cost by Feature and Model
Attribute model spend, tokens, request volume, and cost per request to product features.
Open recipeQuery Nested AI Tool-Call Events
Filter dotted nested fields and rank failing tools without flattening the original event payload.
Open recipeCustomer evidence
How teams use this workflow
Related capabilities
Continue the event-to-decision workflow
AI-assisted analysis
Turn a plain-language question into SQL you can inspect and improve
Explore capabilityStructured event ingestion
Turn application behavior into queryable event tables
Explore capabilitySQL workbench and Query API
Ask detailed questions with the same SQL humans and agents can inspect
Explore capabilityStart with one production workflow
Use a focused prompt, send synthetic events, and verify the first useful query before expanding coverage.