Skip to content
Telemetry
For teams measuring agent outcomes beyond API success

AI agent quality and evaluation

Compare agent task success across releases using reviewer decisions, human handoffs, and run costs.

Reviewed by the Telemetry product team on . We checked the event fields, suggested queries, and data to exclude. Who reviews this page

Connect real telemetryCopy the prompt
Why this works
  • Track whether a run finished separately from whether a reviewer accepted its result.
  • Compare acceptance, revision, rejection, and human-handoff rates.
  • Track cost and latency beside quality instead of optimizing them alone.
What to record and check

Compare agent results before a release

Record the version used in production and how many runs you evaluated. Keep automated scores separate from human reviews so you can check where they disagree.

  1. 1

    Versioned run

    Record workflow, model, prompt, tools, release, cost, and terminal status.

  2. 2

    Reviewed outcome

    Attach a rubric version, evaluator type, score, and acceptance or handoff result.

  3. 3

    SQL comparison

    Compare quality, coverage, failure, latency, and cost across the same eligible population.

  4. 4

    Release decision

    Inspect failed examples and product impact before promoting or rolling back.

Use case versus template

Choose what to measure

Use this guide to choose what to measure and when to log it. For a shorter setup prompt, open the matching template.

Open AI agent observability template

Agent prompt

Paste this into your coding agent

Replace YOUR_API_KEY after signup, then ask the agent to run the product flow and verify the first events.

agent prompt

AI agent quality and evaluation setup prompt

text
Instrument AI agent quality and evaluation with Telemetry.

Use /skill.md and this Telemetry API key: YOUR_API_KEY

Log terminal agent outcomes with run_id, workflow, model, prompt_version, release, status, reviewer_outcome, human_handoff, duration_ms, input_tokens, output_tokens, estimated_cost_usd, retry_count, and controlled error_type. Log tool outcomes separately with a shared run_id.

Create dashboards for technical success, reviewed acceptance, revision and rejection, human handoff, cost per accepted outcome, and quality by release. Exclude unreviewed runs from the reviewed-acceptance denominator.

Do not log raw prompts, completions, tool payloads, secrets, or personal data unless the data owner has explicitly approved the purpose and retention.

Setup steps

  1. 1Write down what counts as a successful task and which verdicts reviewers can choose.
  2. 2Log how each agent run ends. Add events for tool calls and human handoffs.
  3. 3Allow time for reviews to finish, then group reviewed runs by workflow, model, prompt, and release.
  4. 4Inspect examples in your review system to understand a quality drop before changing the agent.

Events to capture

agent_run_completedagent_output_reviewedagent_handoff_requestedagent_tool_completedagent_release_evaluated

Questions you can answer

  • Which workflows complete technically but fail human review?
  • Where is human handoff increasing after a release?
  • Which models produce accepted results within your budget?

Example event schemas

Event schemas for this workflow

Check what each event records, when to send it, and which field types it needs. Review the example payload and privacy checklist before using it in production.

AI operations10 fields

llm_request_completed

One completed model-provider request.

View schema
AI agent operations10 fields

ai_agent_run_completed

One terminal outcome per logical agent run.

View schema
AI operations11 fields

agent_tool_call_completed

One completed tool-call attempt within an agent run.

View schema

Use these queries in Telemetry

Learn about AI agent monitoring

Query agent events to compare tool use, model costs, and outcomes for each run.

Explore AI agent monitoringRead customer stories

Related SQL recipes

More SQL recipes

Run the query using this workflow's event fields and check the example result. Save the result to a dashboard or set up an alert.

Browse all recipes
AI and LLMIntermediate

Find AI quality regressions by prompt version

Did the new prompt version improve quality without increasing human handoffs?

Open recipe
AI and LLMIntermediate

Measure AI agent task success and human handoff

Which agent workflows finish successfully and produce accepted outcomes?

Open recipe
AI and LLMIntermediate

Measure accepted AI outputs per dollar

Which model and feature combination produces the most accepted outputs per dollar?

Open recipe
AI and LLMIntermediate

Detect repeating AI agent tool loops

Which agent runs appear stuck in a repetitive tool loop?

Open recipe
AI and LLMBeginner

Calculate LLM cost by feature and model

Which product features and models are driving LLM spend?

Open recipe
AI and LLMIntermediate

Measure LLM time to first token

Which model and feature combinations feel slow before output begins?

Open recipe
AI and LLMIntermediate

Evaluate RAG retrieval quality by version

Did the new RAG pipeline improve retrieval and grounded-answer rates?

Open recipe
Recipe collectionsAI and LLM SQL

Next step

Create the API key your agent will use

The free plan is enough to run the prompt, send test events, and review the first dashboard.

Prompt packConnect real telemetry

Related pages

AI agent telemetry and observability

Record agent runs and tool calls in Telemetry tables. Query retries, latency, cost, failures, and whether users accepted the results.

Open page

OpenAI cost monitoring

Track OpenAI and LLM costs by model, feature, and customer. Use SQL dashboards to compare spending with latency, failures, and accepted outputs.

Open page

AI agent security monitoring

Monitor tool authorization, policy denials, approval queues, risky action classes, and release changes without storing prompts, credentials, or tool payloads.

Open page