Skip to content
Telemetry
AI and LLM SQL recipe

Measure AI agent task success and human handoff

Compare agent task success, reviewer acceptance, cost, latency, and human-handoff rate by workflow and model.

Intermediateagent_run_outcomesReviewed 2026-07-28Tested with Apache DataFusion 45.2.0

Reviewed by the Telemetry product team on . We checked the SQL syntax, required event fields, sample results, and limits on using the query. Who reviews this page

Question answered

Which agent workflows finish successfully and produce accepted outcomes?

An agent can finish without solving the task. Compare runtime success with reviewed results and human handoffs to see how often it produces useful work.

Event schema

Fields the query expects

FieldTypeWhy it exists
timestamp_utcTimestampTerminal agent-run time in UTC.
workflowUtf8Stable product workflow or task category.
modelUtf8Model identifier recorded from the provider response.
statusUtf8success or failed for the technical run.
reviewer_outcomeUtf8accepted, revised, rejected, or not_reviewed.
human_handoffBooleanWhether a human took over the task.
duration_msFloat64End-to-end workflow duration.
estimated_cost_usdFloat64Normalized estimated model cost for the run.
DataFusion SQL

Copy the query

sql
SELECT
  workflow,
  model,
  COUNT(*) AS runs,
  100.0 * SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END)
    / NULLIF(COUNT(*), 0) AS technical_success_rate_pct,
  100.0 * SUM(CASE WHEN reviewer_outcome = 'accepted' THEN 1 ELSE 0 END)
    / NULLIF(SUM(CASE
      WHEN reviewer_outcome <> 'not_reviewed' THEN 1 ELSE 0
    END), 0) AS reviewed_acceptance_rate_pct,
  100.0 * SUM(CASE WHEN human_handoff THEN 1 ELSE 0 END)
    / NULLIF(COUNT(*), 0) AS human_handoff_rate_pct,
  AVG(duration_ms) AS average_duration_ms,
  SUM(estimated_cost_usd) / NULLIF(COUNT(*), 0) AS cost_per_run_usd
FROM agent_run_outcomes
WHERE timestamp_utc >= now() - INTERVAL '30 days'
GROUP BY workflow, model
HAVING COUNT(*) >= 20
ORDER BY reviewed_acceptance_rate_pct, runs DESC;

This read-only query is planned and executed against an empty typed table with Apache DataFusion 45.2.0. We review the synthetic sample output separately. Check field types, thresholds, and counting rules against your own data. Read the testing methodology.

Query result

Reviewed agent acceptance by workflow

Refund review completes technically but needs substantially more revision or human takeover than the other workflows.

workflowmodelrunstechnical_success_rate_pctreviewed_acceptance_rate_pcthuman_handoff_rate_pctaverage_duration_mscost_per_run_usd
support_triageconfigured-model-a1,84098.186.418.24,8200.04
refund_reviewconfigured-model-b62096.868.742.98,1100.07
knowledge_draftconfigured-model-a94099.291.68.43,9100.04

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

Reviewed agent acceptance by workflow: static chart of synthetic reviewed_acceptance_rate_pct values from the Measure AI agent task success and human handoff example result
Download this SVG chart of the sample results for an article, runbook, or design review. Please credit Telemetry.

Reproduce the example

Download the sample data

The JSON bundle includes the event schema with field types, illustrative input rows, exact SQL, expected output, review notes, and engine version. The CSV contains the displayed result.

How the SQL works

  1. 1Technical success and reviewed acceptance remain separate so a completed API call cannot stand in for product quality.
  2. 2The acceptance denominator excludes unreviewed runs rather than silently treating missing labels as rejection.
  3. 3Handoff, duration, and cost expose the operational tradeoff behind an apparently high completion rate.

Edge cases to check

  • Reviewer outcomes need a stable rubric and inter-rater checks before comparing teams or model versions.
  • Automatically resolved easy tasks can bias acceptance upward; segment by task difficulty when it changes.
  • Keep raw prompts, completions, and sensitive tool results out of the event unless explicitly approved.

Recommended dashboard

  • Bars: reviewed acceptance and human handoff by workflow
  • Trend: acceptance after model or prompt releases
  • Table: runs grouped by controlled reviewer outcome

Alert guidance

Compare cohorts after their review windows close. Investigate lower acceptance or more handoffs after a release. Page someone only for user-facing failures or safety violations.

Read alert setup

Set up the events this query needs

Related instrumentation and guides

Define the source data

Event schemas for this analysis

Continue the analysis

Run it on your events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key