Evaluate AI agents with structured events and SQL

An agent can finish a run and still fail the task. Record the run, the model and tool calls you need to analyze, and the evaluation result as separate events. SQL can compare quality, cost, latency, retries, and human handoffs by release. Check model-generated scores against other evidence before treating them as correct.

Telemetry is the analysis layer in this workflow. It does not execute scorers, manage prompt versions, curate evaluation datasets, or provide prompt-and-completion replay. Keep those workflows in your application or a dedicated evaluation system, then send the approved outcome fields needed for aggregate analysis.

Define the unit of evaluation first

Choose the row that answers "what received a score?" before choosing metrics. Common units include:

• one final answer shown to a user;
• one completed agent run;
• one resolved support case;
• one tool selection decision;
• one retrieved answer against a frozen example;
• one business operation that can include several agent attempts.

Give that unit a globally unique identifier such as operation_id. Use separate identifiers for run_id, response_id, and evaluation_id. A retry can create several runs for one operation, and one output can receive several evaluations. Reusing a single identifier for every grain produces incorrect joins and double-counted cost. For a release-level pass rate, decide which review is authoritative before counting the operation.

Separate run, request, and evaluation events

A practical starting contract uses three or four event tables:

Do not force all four grains into one wide row. A run with five tool attempts and two evaluations would otherwise multiply costs or success counts when joined.

Use several kinds of evidence

No single evaluator is sufficient for every agent workflow. Combine only the signals that correspond to an actual decision:

1. Deterministic checks verify schemas, required citations, allowed tool choices, exact calculations, policy rules, or known terminal state.
2. Human review captures a bounded rubric such as correct, partially correct, unsafe, or needs escalation. Record the rubric and reviewer process, not private reviewer notes.
3. Model-based scoring can apply a repeatable rubric at higher volume. Version the judge model, instructions, and thresholds, and periodically compare the score with human review.
4. Product outcomes record whether the user accepted, saved, corrected, regenerated, escalated, or abandoned the result.

An LLM judge is a measurement instrument, not an objective label. Track disagreement, missing evaluations, and changes in judge configuration. The Langfuse evaluation concepts and Arize Phoenix evaluation documentation describe additional evaluation workflows that can remain upstream of Telemetry.

Emit a reviewed outcome event

This JavaScript example sends a compact evaluator result after the scorer or human-review workflow has finished:

import telemetry from "telemetry-sh";

telemetry.init(process.env.TELEMETRY_API_KEY);

export async function recordAgentEvaluation({
  operationId,
  evaluationId,
  evaluatorType,
  evaluatorVersion,
  metricName,
  score,
  threshold,
  reviewOutcome,
  isFinal,
  datasetVersion,
  promptVersion,
  release,
}) {
  await telemetry.log("ai_output_reviewed", {
    operation_id: operationId,
    evaluation_id: evaluationId,
    evaluator_type: evaluatorType,
    evaluator_version: evaluatorVersion,
    metric_name: metricName,
    score,
    threshold,
    passed: score >= threshold,
    review_outcome: reviewOutcome,
    is_final: isFinal,
    dataset_version: datasetVersion,
    prompt_version: promptVersion,
    release,
  });
}

Set isFinal to true only for the one decision your workflow treats as authoritative under that operation, metric, evaluator version, and release. A completed agent run does not by itself finalize a review.

Keep raw prompts, completions, retrieved documents, tool arguments, secrets, and free-form reviewer notes out of the event by default. Prefer stable categories and version identifiers. If content retention is approved, store it in the system designed for that access and deletion policy and correlate it with a restricted identifier.

Compare quality by release

This example treats one is_final = true review for each operation_id, release, metric, and evaluator version as authoritative. Emit preliminary and independent reviews with is_final = false; if your workflow can revise a final decision, use a new evaluator version or a separate finalization event instead of emitting a second final row under the same key. The query excludes keys with duplicate final rows. Investigate those conflicts rather than letting an earlier pass override a later failure. It also limits the reviewed denominator to completed operations. The explicit evaluation count prevents an unevaluated release from looking artificially successful.

WITH completed AS (
  SELECT release, operation_id
  FROM agent_run_completed
  WHERE timestamp_utc >= now() - INTERVAL '30 days'
    AND status = 'success'
  GROUP BY release, operation_id
),
final_reviews AS (
  SELECT release, operation_id, MAX(score) AS score,
    MAX(CASE WHEN passed THEN 1 ELSE 0 END) AS passed
  FROM ai_output_reviewed
  WHERE timestamp_utc >= now() - INTERVAL '30 days'
    AND metric_name = 'task_quality'
    AND evaluator_version = 'quality-rubric-v3'
    AND is_final = true
  GROUP BY release, operation_id
  HAVING COUNT(*) = 1
)
SELECT
  c.release,
  COUNT(*) AS completed_operations,
  COUNT(f.operation_id) AS evaluated_operations,
  ROUND(
    100.0 * COUNT(f.operation_id)
    / NULLIF(COUNT(*), 0),
    1
  ) AS evaluation_coverage_pct,
  ROUND(
    100.0 * SUM(CASE WHEN f.passed = 1 THEN 1 ELSE 0 END)
    / NULLIF(COUNT(f.operation_id), 0),
    1
  ) AS evaluated_pass_rate_pct,
  ROUND(AVG(f.score), 3) AS average_final_score
FROM completed c
LEFT JOIN final_reviews f
  ON f.release = c.release AND f.operation_id = c.operation_id
GROUP BY c.release
ORDER BY c.release;

Check for the final-review conflicts excluded from that result before using it for a release decision:

SELECT release, operation_id, COUNT(*) AS conflicting_final_reviews
FROM ai_output_reviewed
WHERE timestamp_utc >= now() - INTERVAL '30 days'
  AND metric_name = 'task_quality'
  AND evaluator_version = 'quality-rubric-v3'
  AND is_final = true
GROUP BY release, operation_id
HAVING COUNT(*) > 1
ORDER BY release, operation_id;

Do not compare releases that used different rubrics, judge models, thresholds, dataset versions, or sampling rules without separating those dimensions. A score change after an evaluator change is not evidence of a product regression.

Calculate cost per accepted outcome

Aggregate provider cost to the operation grain before joining it to one authoritative terminal outcome. The globally unique operation_id in this example spans releases; if yours does not, include release in both the request and outcome keys. This query excludes conflicting final outcomes, which should be investigated separately.

WITH operation_cost AS (
  SELECT
    operation_id,
    SUM(estimated_cost_usd) AS total_cost_usd
  FROM llm_request_completed
  WHERE timestamp_utc >= now() - INTERVAL '30 days'
  GROUP BY operation_id
),
terminal_outcome AS (
  SELECT
    operation_id,
    MAX(CASE WHEN review_outcome = 'accepted' THEN 1 ELSE 0 END) AS accepted
  FROM ai_output_reviewed
  WHERE timestamp_utc >= now() - INTERVAL '30 days'
    AND metric_name = 'task_quality'
    AND evaluator_version = 'quality-rubric-v3'
    AND is_final = true
  GROUP BY operation_id
  HAVING COUNT(*) = 1
)
SELECT
  COUNT(*) AS evaluated_operations,
  SUM(accepted) AS accepted_operations,
  ROUND(SUM(total_cost_usd), 4) AS evaluated_cost_usd,
  ROUND(
    SUM(total_cost_usd) / NULLIF(SUM(accepted), 0),
    4
  ) AS cost_per_accepted_operation_usd
FROM operation_cost c
JOIN terminal_outcome o ON o.operation_id = c.operation_id;

This metric is only meaningful when "accepted" has a stable definition. A copied answer, a user-visible answer, a case resolved without reopening, and a human rubric pass are different outcomes.

Build a regression gate

For each proposed release:

1. Run the same frozen dataset with the same evaluator configuration.
2. Record the candidate release, prompt_version, dataset_version, and evaluator_version.
3. Compare pass rate, severe-failure rate, handoff rate, p95 duration, and cost per accepted operation with the approved baseline.
4. Inspect failed examples in the source evaluation system.
5. Approve or reject the release using thresholds chosen before the run.
6. Monitor production outcomes separately because a frozen dataset cannot represent every live input.

Include minimum sample sizes and confidence intervals when the decision warrants them. Avoid an alert on a percentage with a tiny denominator. Also track evaluation coverage: a passing score over 20 reviewed runs does not describe 10,000 unreviewed runs.

When to retain a dedicated evaluation platform

Use a specialist platform when the team needs prompt and completion inspection, dataset curation, annotation queues, prompt management, experiment execution, trace replay, or built-in evaluators. Telemetry can receive the resulting versioned scores and outcomes for SQL analysis; it is not a feature-for-feature replacement.

Next, use the AI quality regression recipe, agent task success and handoff recipe, and accepted output per dollar recipe. For the wider implementation boundary, see AI agent monitoring.