Record the result where the task finishes
Use the model SDK guide for request cost and latency, then add the agent or framework guide where tool selection, retries, and final task status become known.
Selection framework
Log where the final result is known
Keep traces and outcomes distinct
Keep detailed spans in your trace backend. Send the final result to Telemetry with an approved identifier to link it to the trace.
Measure value beside cost
Connect tokens and latency to reviewed outcomes such as accepted, saved, resolved, escalated, or handed off instead of optimizing request volume alone.
Shared event contract
Use the same field definitions across services
- run_id, feature, provider, model, and status
- input_tokens, output_tokens, estimated_cost_usd, and latency_ms
- tool_name, retry_count, cache_hit, reviewer_outcome, and human_handoff
Adapt these fields to your workflow. Document units, allowed statuses, and identifiers before several services write to the same table. Name the team responsible for the schema.
Verification
Exercise failure paths before alerts
- 1Run one successful model request and reconcile provider-reported tokens.
- 2Exercise one tool failure, retry, and final agent outcome.
- 3Confirm raw prompts, completions, credentials, and private tool payloads are excluded.
- 4Query cost, latency, failure rate, and accepted outcomes by feature.
Implementation guides
Match the framework, provider, or runtime
Setup guide with source links
OpenAI agent telemetry
Log OpenAI agent runs, tool calls, model usage, latency, cost, and final outcomes with structured events.
Open guideSetup guide with source links
OpenAI Responses API telemetry
Measure OpenAI Responses API latency, token usage, tool activity, failures, and downstream outcomes without collecting prompts or generated content.
Open guideSetup guide with source links
LiteLLM gateway and SDK telemetry
Track LiteLLM provider routing, retries, fallbacks, token usage, latency, cost estimates, and accepted outcomes across a controlled model gateway.
Open guideSetup guide with source links
Anthropic Claude API telemetry
Track Claude API token usage, model, latency, stop reason, errors, cost context, and reviewed product outcomes without storing prompts.
Open guideSetup guide with source links
Google Gemini API telemetry
Track Gemini request latency, model usage, tokens, finish reason, errors, and reviewed product outcomes without storing prompts or generated content.
Open guideSetup guide with source links
Amazon Bedrock model telemetry
Measure Bedrock Converse requests by model or inference profile, token usage, latency, stop reason, retries, and business outcome.
Open guideSetup guide with source links
Azure OpenAI Responses telemetry
Track Azure OpenAI deployment usage, token cost, latency, failures, and product outcomes with editable SQL-ready events.
Open guideSetup guide with source links
Vercel AI SDK telemetry
Track streaming completions, token usage, retries, latency, and accepted results from AI SDK workflows.
Open guideSetup guide with source links
LangChain agent outcome telemetry
Connect LangChain agent runs and tool outcomes to product-facing cost, reliability, handoff, and acceptance metrics without duplicating raw traces.
Open guideSetup guide with source links
LangGraph agent observability
Measure LangGraph run outcomes, node and tool failures, retries, checkpoints, human handoffs, latency, and reviewed task quality.
Open guideSetup guide with source links
Claude Agent SDK telemetry
Track Claude Agent SDK run outcomes, duration, tool activity, turns, cost, and approved product signals without storing prompts or tool payloads.
Open guideSetup guide with source links
Pydantic AI agent telemetry
Measure Pydantic AI run outcomes, validated outputs, retries, tool activity, latency, usage, and product acceptance with safe structured events.
Open guideSetup guide with source links
CrewAI workflow telemetry
Track CrewAI crew and flow outcomes, task counts, handoffs, retries, latency, cost, and accepted results without retaining agent conversations.
Open guideSetup guide with source links
Google ADK agent telemetry
Measure Google Agent Development Kit sessions, run outcomes, tool activity, handoffs, latency, and approved product signals with SQL-ready events.
Open guideSetup guide with source links
Mastra agent telemetry
Track Mastra agent and workflow outcomes, tool activity, latency, retries, cost, and product acceptance alongside built-in traces and evals.
Open guideSetup guide with source links
LlamaIndex agent telemetry
Track LlamaIndex agent and workflow outcomes, tool activity, latency, retrieval versions, cost, and reviewed quality with structured events.
Open guideSetup guide with source links
DSPy program evaluation telemetry
Measure DSPy program latency, failures, metric scores, optimizer versions, model cost, and accepted outcomes across releases with SQL-ready events.
Open guideSetup guide with source links
AutoGen task flow, state, and timeout monitoring
Track AutoGen task-flow state, timeouts, team and agent outcomes, tool activity, handoffs, latency, failures, cost, and releases with compact structured events.
Open guideSetup guide with source links
Semantic Kernel agent telemetry
Measure Semantic Kernel function and agent outcomes, tool activity, model usage, latency, cost, and releases alongside OpenTelemetry traces.
Open guideSetup guide with source links
MCP server and tool telemetry
Track Model Context Protocol tool outcomes, latency, errors, approvals, retries, clients, and releases without storing arguments or returned content.
Open guideVerify one end-to-end workflow first
Create a free API key, choose the closest runtime guide, and send one known success and failure before expanding coverage.