Telemetry
For teams measuring agent outcomes beyond API success

AI Agent Quality and Evaluation

Connect agent runs, reviewer outcomes, human handoffs, cost, and release changes to a repeatable quality-evaluation workflow.

Reviewed by the Telemetry product team on . Event contract, recommended analysis, and privacy boundaries. Editorial ownership

Get an API keyCopy the prompt
Why this works
  • Keep technical completion separate from reviewed task success.
  • Compare acceptance, revision, rejection, and human-handoff rates.
  • Track cost and latency beside quality instead of optimizing them alone.

Agent prompt

Paste this into your coding agent

Replace YOUR_API_KEY after signup, then ask the agent to run the product flow and verify the first events.

agent prompt

AI Agent Quality and Evaluation setup prompt

text
Instrument AI agent quality and evaluation with Telemetry.

Use /skill.md and this Telemetry API key: YOUR_API_KEY

Log terminal agent outcomes with run_id, workflow, model, prompt_version, release, status, reviewer_outcome, human_handoff, duration_ms, input_tokens, output_tokens, estimated_cost_usd, retry_count, and controlled error_type. Log tool outcomes separately with a shared run_id.

Create dashboards for technical success, reviewed acceptance, revision and rejection, human handoff, cost per accepted outcome, and quality by release. Exclude unreviewed runs from the reviewed-acceptance denominator.

Do not log raw prompts, completions, tool payloads, secrets, or personal data unless the data owner has explicitly approved the purpose and retention.

Setup steps

  1. 1Define a task-level success rubric and controlled reviewer outcomes.
  2. 2Log one terminal outcome per agent run plus tool and handoff events.
  3. 3Segment mature reviewed cohorts by workflow, model, prompt, and release.
  4. 4Review regressions with examples outside telemetry before changing automation.

Events to capture

agent_run_completedagent_output_reviewedagent_handoff_requestedagent_tool_completedagent_release_evaluated

Questions unlocked

  • Which workflows complete technically but fail human review?
  • Where is human handoff increasing after a release?
  • Which model produces accepted outcomes at a sustainable cost?

Related SQL recipes

Answer the next question with SQL

Run the query against the structured fields from this workflow, inspect the example result, and turn a useful answer into a dashboard or alert.

Browse all recipes
AI and LLMIntermediate

Find AI Quality Regressions by Prompt Version

Did the new prompt version improve quality without increasing human handoffs?

Open recipe
AI and LLMIntermediate

Measure AI Agent Task Success and Human Handoff

Which agent workflows finish successfully and produce accepted outcomes?

Open recipe
AI and LLMIntermediate

Measure Accepted AI Outputs per Dollar

Which model and feature combination produces the most accepted outputs per dollar?

Open recipe
AI and LLMIntermediate

Detect Repeating AI Agent Tool Loops

Which agent runs appear stuck in a repetitive tool loop?

Open recipe
AI and LLMBeginner

Calculate LLM Cost by Feature and Model

Which product features and models are driving LLM spend?

Open recipe
AI and LLMIntermediate

Measure LLM Time to First Token

Which model and feature combinations feel slow before output begins?

Open recipe

Next step

Create the API key your agent will use

The free plan is enough to run the prompt, send test events, and review the first dashboard.

Prompt packStart free

Related pages

Claude Code Observability

Give Claude Code a prompt that makes telemetry part of the implementation pass instead of a separate cleanup project.

Open page

Cursor Telemetry Setup

A copyable setup prompt for adding event tables, funnel tracking, and dashboards from inside Cursor.

Open page

Codex Instrumentation Prompt

A focused prompt that asks Codex to instrument the rest of the product, verify events, and summarize coverage gaps.

Open page