Instrument the owner of the final outcome
Use the model SDK guide for request cost and latency, then add the agent or framework guide where tool selection, retries, and final task status become known.
Selection framework
Choose the boundary that knows the result
Keep traces and outcomes distinct
Retain a trace backend for detailed spans when needed. Send a bounded product outcome to Telemetry with an approved correlation identifier.
Measure value beside cost
Connect tokens and latency to reviewed outcomes such as accepted, saved, resolved, escalated, or handed off instead of optimizing request volume alone.
Shared event contract
Keep the first schema interoperable
- run_id, feature, provider, model, and status
- input_tokens, output_tokens, estimated_cost_usd, and latency_ms
- tool_name, retry_count, cache_hit, reviewer_outcome, and human_handoff
Adapt these fields to the workflow, but keep units, status categories, identifiers, and ownership explicit before several services depend on the same table.
Verification
Exercise failure paths before alerts
- 1Run one successful model request and reconcile provider-reported tokens.
- 2Exercise one tool failure, retry, and final agent outcome.
- 3Confirm raw prompts, completions, credentials, and private tool payloads are excluded.
- 4Query cost, latency, failure rate, and accepted outcomes by feature.
Implementation guides
Match the framework, provider, or runtime
Sourced implementation guide
OpenAI Agent Telemetry
Log OpenAI agent runs, tool calls, model usage, latency, cost, and final outcomes with structured events.
Open guideSourced implementation guide
OpenAI Responses API Telemetry
Measure OpenAI Responses API latency, token usage, tool activity, failures, and downstream outcomes without collecting prompts or generated content.
Open guideSourced implementation guide
LiteLLM Gateway and SDK Telemetry
Track LiteLLM provider routing, retries, fallbacks, token usage, latency, cost estimates, and accepted outcomes across a controlled model gateway.
Open guideSourced implementation guide
Anthropic Claude API Telemetry
Track Claude API token usage, model, latency, stop reason, errors, cost context, and reviewed product outcomes without storing prompts.
Open guideSourced implementation guide
Google Gemini API Telemetry
Track Gemini request latency, model usage, tokens, finish reason, errors, and reviewed product outcomes without storing prompts or generated content.
Open guideSourced implementation guide
Amazon Bedrock Model Telemetry
Measure Bedrock Converse requests by model or inference profile, token usage, latency, stop reason, retries, and business outcome.
Open guideSourced implementation guide
Azure OpenAI Responses Telemetry
Track Azure OpenAI deployment usage, token cost, latency, failures, and product outcomes with editable SQL-ready events.
Open guideSourced implementation guide
Vercel AI SDK Telemetry
Track streaming completions, token usage, retries, latency, and accepted results from AI SDK workflows.
Open guideSourced implementation guide
LangChain Agent Outcome Telemetry
Connect LangChain agent runs and tool outcomes to product-facing cost, reliability, handoff, and acceptance metrics without duplicating raw traces.
Open guideSourced implementation guide
LangGraph Agent Observability
Measure LangGraph run outcomes, node and tool failures, retries, checkpoints, human handoffs, latency, and reviewed task quality.
Open guideSourced implementation guide
Claude Agent SDK Telemetry
Track Claude Agent SDK run outcomes, duration, tool activity, turns, cost, and approved product signals without storing prompts or tool payloads.
Open guideSourced implementation guide
Pydantic AI Agent Telemetry
Measure Pydantic AI run outcomes, validated outputs, retries, tool activity, latency, usage, and product acceptance with safe structured events.
Open guideSourced implementation guide
CrewAI Workflow Telemetry
Track CrewAI crew and flow outcomes, task counts, handoffs, retries, latency, cost, and accepted results without retaining agent conversations.
Open guideSourced implementation guide
Google ADK Agent Telemetry
Measure Google Agent Development Kit sessions, run outcomes, tool activity, handoffs, latency, and approved product signals with SQL-ready events.
Open guideSourced implementation guide
Mastra Agent Telemetry
Track Mastra agent and workflow outcomes, tool activity, latency, retries, cost, and product acceptance alongside built-in traces and evals.
Open guideSourced implementation guide
LlamaIndex Agent Telemetry
Track LlamaIndex agent and workflow outcomes, tool activity, latency, retrieval versions, cost, and reviewed quality with structured events.
Open guideSourced implementation guide
DSPy Program Evaluation Telemetry
Measure DSPy program latency, failures, metric scores, optimizer versions, model cost, and accepted outcomes across releases with SQL-ready events.
Open guideSourced implementation guide
AutoGen Multi-Agent Telemetry
Track AutoGen team and agent outcomes, messages, tool activity, handoffs, latency, failures, cost, and releases with compact structured events.
Open guideSourced implementation guide
Semantic Kernel Agent Telemetry
Measure Semantic Kernel function and agent outcomes, tool activity, model usage, latency, cost, and releases alongside OpenTelemetry traces.
Open guideSourced implementation guide
MCP Server and Tool Telemetry
Track Model Context Protocol tool outcomes, latency, errors, approvals, retries, clients, and releases without storing arguments or returned content.
Open guideVerify one end-to-end workflow first
Create a free API key, choose the closest runtime guide, and send one known success and failure before expanding coverage.