Skip to content
Telemetry
For teams measuring agent outcomes beyond API success

AI Agent Quality and Evaluation

Connect agent runs, reviewer outcomes, human handoffs, cost, and release changes to a repeatable quality-evaluation workflow.

Reviewed by the Telemetry product team on . Event contract, recommended analysis, and privacy boundaries. Review standards and ownership

Start with sample dataCopy the prompt
Why this works
  • Keep technical completion separate from reviewed task success.
  • Compare acceptance, revision, rejection, and human-handoff rates.
  • Track cost and latency beside quality instead of optimizing them alone.
Measurement path

Move from one agent run to a release decision

A useful quality loop preserves versioned production context, separates automated scores from reviewed outcomes, and keeps evaluated coverage visible.

  1. 1

    Versioned run

    Record workflow, model, prompt, tools, release, cost, and terminal status.

  2. 2

    Reviewed outcome

    Attach a rubric version, evaluator type, score, and acceptance or handoff result.

  3. 3

    SQL comparison

    Compare quality, coverage, failure, latency, and cost across the same eligible population.

  4. 4

    Release decision

    Inspect failed examples and product impact before promoting or rolling back.

Use case versus template

This page explains what to measure and why

Use the use-case guide to choose outcomes, event boundaries, and analysis questions. Open the matching template when you are ready for a shorter copy-paste implementation brief.

Open AI Agent Observability Template

Agent prompt

Paste this into your coding agent

Replace YOUR_API_KEY after signup, then ask the agent to run the product flow and verify the first events.

agent prompt

AI Agent Quality and Evaluation setup prompt

text
Instrument AI agent quality and evaluation with Telemetry.

Use /skill.md and this Telemetry API key: YOUR_API_KEY

Log terminal agent outcomes with run_id, workflow, model, prompt_version, release, status, reviewer_outcome, human_handoff, duration_ms, input_tokens, output_tokens, estimated_cost_usd, retry_count, and controlled error_type. Log tool outcomes separately with a shared run_id.

Create dashboards for technical success, reviewed acceptance, revision and rejection, human handoff, cost per accepted outcome, and quality by release. Exclude unreviewed runs from the reviewed-acceptance denominator.

Do not log raw prompts, completions, tool payloads, secrets, or personal data unless the data owner has explicitly approved the purpose and retention.

Setup steps

  1. 1Define a task-level success rubric and controlled reviewer outcomes.
  2. 2Log one terminal outcome per agent run plus tool and handoff events.
  3. 3Segment mature reviewed cohorts by workflow, model, prompt, and release.
  4. 4Review regressions with examples outside telemetry before changing automation.

Events to capture

agent_run_completedagent_output_reviewedagent_handoff_requestedagent_tool_completedagent_release_evaluated

Questions unlocked

  • Which workflows complete technically but fail human review?
  • Where is human handoff increasing after a release?
  • Which model produces accepted outcomes at a sustainable cost?

Event schema starting points

Event contracts for this workflow

Review the row grain, emit boundary, required types, privacy classes, example payload, and validation checklist before adapting a query or snippet to production.

AI operations10 fields

llm_request_completed

One completed model-provider request.

Inspect contract
AI agent operations10 fields

ai_agent_run_completed

One terminal outcome per logical agent run.

Inspect contract
AI operations11 fields

agent_tool_call_completed

One completed tool-call attempt within an agent run.

Inspect contract

Related product capability

Continue this workflow in AI agent monitoring

Connect agent runs, tool use, model cost, quality, and product outcomes with reviewable SQL.

Explore AI agent monitoringReview customer evidence

Related SQL recipes

Answer the next question with SQL

Run the query against the structured fields from this workflow, inspect the example result, and turn a useful answer into a dashboard or alert.

Browse all recipes
AI and LLMIntermediate

Find AI Quality Regressions by Prompt Version

Did the new prompt version improve quality without increasing human handoffs?

Open recipe
AI and LLMIntermediate

Measure AI Agent Task Success and Human Handoff

Which agent workflows finish successfully and produce accepted outcomes?

Open recipe
AI and LLMIntermediate

Measure Accepted AI Outputs per Dollar

Which model and feature combination produces the most accepted outputs per dollar?

Open recipe
AI and LLMIntermediate

Detect Repeating AI Agent Tool Loops

Which agent runs appear stuck in a repetitive tool loop?

Open recipe
AI and LLMBeginner

Calculate LLM Cost by Feature and Model

Which product features and models are driving LLM spend?

Open recipe
AI and LLMIntermediate

Measure LLM Time to First Token

Which model and feature combinations feel slow before output begins?

Open recipe
AI and LLMIntermediate

Evaluate RAG Retrieval Quality by Version

Did the new RAG pipeline improve retrieval and grounded-answer rates?

Open recipe
Complete collectionsAI and LLM SQL

Next step

Create the API key your agent will use

The free plan is enough to run the prompt, send test events, and review the first dashboard.

Prompt packStart with sample data

Related pages

AI Agent Telemetry and Observability

Instrument AI agent telemetry for runs, tool calls, retries, latency, model cost, failures, and accepted outcomes in SQL-ready event tables.

Open page

OpenAI Cost Monitoring

Monitor OpenAI and LLM spend by model, feature, customer, latency, error rate, and outcome with SQL dashboards and budget alerts.

Open page

AI Agent Security Monitoring

Monitor tool authorization, policy denials, approval queues, risky action classes, and release changes without storing prompts, credentials, or tool payloads.

Open page