Evaluate AI Agents with Structured Events and SQL

AI agent evaluation is useful when it distinguishes a technically completed run from a result that actually satisfied the task. A reliable design records compact events for the run, any model or tool activity worth analyzing, and the later evaluation outcome. SQL can then compare quality, cost, latency, retries, and human handoffs by release without treating a trace or a model-generated score as ground truth.

Telemetry is the analysis layer in this workflow. It does not execute scorers, manage prompt versions, curate evaluation datasets, or provide prompt-and-completion replay. Keep those workflows in your application or a dedicated evaluation system, then send the approved outcome fields needed for aggregate analysis.

Define the unit of evaluation first

Choose the row that answers “what received a score?” before choosing metrics. Common units include:

• one final answer shown to a user;
• one completed agent run;
• one resolved support case;
• one tool selection decision;
• one retrieved answer against a frozen example;
• one business operation that can include several agent attempts.

Give that unit a stable identifier such as operation_id. Use separate identifiers for run_id, response_id, and evaluation_id. A retry can create several runs for one operation, and one output can receive several evaluations. Reusing a single identifier for every grain produces incorrect joins and double-counted cost.

Separate run, request, and evaluation events

A practical starting contract uses three or four event tables:

Do not force all four grains into one wide row. A run with five tool attempts and two evaluations would otherwise multiply costs or success counts when joined.

Use several kinds of evidence

No single evaluator is sufficient for every agent workflow. Combine only the signals that correspond to an actual decision:

1. Deterministic checks verify schemas, required citations, allowed tool choices, exact calculations, policy rules, or known terminal state.
2. Human review captures a bounded rubric such as correct, partially correct, unsafe, or needs escalation. Record the rubric and reviewer process, not private reviewer notes.
3. Model-based scoring can apply a repeatable rubric at higher volume. Version the judge model, instructions, and thresholds, and periodically compare the score with human review.
4. Product outcomes record whether the user accepted, saved, corrected, regenerated, escalated, or abandoned the result.

An LLM judge is a measurement instrument, not an objective label. Track disagreement, missing evaluations, and changes in judge configuration. The Langfuse evaluation concepts and Arize Phoenix evaluation documentation describe additional evaluation workflows that can remain upstream of Telemetry.

Emit a reviewed outcome event

This JavaScript example sends a compact evaluator result after the scorer or human-review workflow has finished:

import telemetry from "telemetry-sh";

telemetry.init(process.env.TELEMETRY_API_KEY);

export async function recordAgentEvaluation({
  operationId,
  evaluationId,
  evaluatorType,
  evaluatorVersion,
  metricName,
  score,
  threshold,
  reviewOutcome,
  datasetVersion,
  promptVersion,
  release,
}) {
  await telemetry.log("ai_output_reviewed", {
    operation_id: operationId,
    evaluation_id: evaluationId,
    evaluator_type: evaluatorType,
    evaluator_version: evaluatorVersion,
    metric_name: metricName,
    score,
    threshold,
    passed: score >= threshold,
    review_outcome: reviewOutcome,
    dataset_version: datasetVersion,
    prompt_version: promptVersion,
    release,
  });
}

Keep raw prompts, completions, retrieved documents, tool arguments, secrets, and free-form reviewer notes out of the event by default. Prefer stable categories and version identifiers. If content retention is approved, store it in the system designed for that access and deletion policy and correlate it with a restricted identifier.

Compare quality by release

This query calculates coverage and pass rate for a single metric. The explicit evaluation count prevents an unevaluated release from looking artificially successful.

WITH run_counts AS (
  SELECT
    release,
    COUNT(DISTINCT operation_id) AS completed_operations
  FROM agent_run_completed
  WHERE timestamp_utc >= now() - INTERVAL '30 days'
    AND status = 'success'
  GROUP BY release
),
evaluation_counts AS (
  SELECT
    release,
    COUNT(DISTINCT operation_id) AS evaluated_operations,
    COUNT(DISTINCT CASE WHEN passed THEN operation_id END) AS passed_operations,
    AVG(score) AS average_score
  FROM ai_output_reviewed
  WHERE timestamp_utc >= now() - INTERVAL '30 days'
    AND metric_name = 'task_quality'
    AND evaluator_version = 'quality-rubric-v3'
  GROUP BY release
)
SELECT
  r.release,
  r.completed_operations,
  COALESCE(e.evaluated_operations, 0) AS evaluated_operations,
  ROUND(
    100.0 * COALESCE(e.evaluated_operations, 0)
    / NULLIF(r.completed_operations, 0),
    1
  ) AS evaluation_coverage_pct,
  ROUND(
    100.0 * COALESCE(e.passed_operations, 0)
    / NULLIF(e.evaluated_operations, 0),
    1
  ) AS evaluated_pass_rate_pct,
  ROUND(e.average_score, 3) AS average_score
FROM run_counts r
LEFT JOIN evaluation_counts e ON e.release = r.release
ORDER BY r.release;

Do not compare releases that used different rubrics, judge models, thresholds, dataset versions, or sampling rules without separating those dimensions. A score change after an evaluator change is not evidence of a product regression.

Calculate cost per accepted outcome

Aggregate provider cost to the operation grain before joining it to a terminal outcome:

WITH operation_cost AS (
  SELECT
    operation_id,
    SUM(estimated_cost_usd) AS total_cost_usd
  FROM llm_request_completed
  WHERE timestamp_utc >= now() - INTERVAL '30 days'
  GROUP BY operation_id
),
terminal_outcome AS (
  SELECT
    operation_id,
    MAX(CASE WHEN review_outcome = 'accepted' THEN 1 ELSE 0 END) AS accepted
  FROM ai_output_reviewed
  WHERE timestamp_utc >= now() - INTERVAL '30 days'
  GROUP BY operation_id
)
SELECT
  COUNT(*) AS evaluated_operations,
  SUM(accepted) AS accepted_operations,
  ROUND(SUM(total_cost_usd), 4) AS evaluated_cost_usd,
  ROUND(
    SUM(total_cost_usd) / NULLIF(SUM(accepted), 0),
    4
  ) AS cost_per_accepted_operation_usd
FROM operation_cost c
JOIN terminal_outcome o ON o.operation_id = c.operation_id;

This metric is only meaningful when “accepted” has a stable definition. A copied answer, a user-visible answer, a case resolved without reopening, and a human rubric pass are different outcomes.

Build a regression gate

For each proposed release:

1. Run the same frozen dataset with the same evaluator configuration.
2. Record the candidate release, prompt_version, dataset_version, and evaluator_version.
3. Compare pass rate, severe-failure rate, handoff rate, p95 duration, and cost per accepted operation with the approved baseline.
4. Inspect failed examples in the source evaluation system.
5. Approve or reject the release using thresholds chosen before the run.
6. Monitor production outcomes separately because a frozen dataset cannot represent every live input.

Include minimum sample sizes and confidence intervals when the decision warrants them. Avoid an alert on a percentage with a tiny denominator. Also track evaluation coverage: a passing score over 20 reviewed runs does not describe 10,000 unreviewed runs.

When to retain a dedicated evaluation platform

Use a specialist platform when the team needs prompt and completion inspection, dataset curation, annotation queues, prompt management, experiment execution, trace replay, or built-in evaluators. Telemetry can receive the resulting versioned scores and outcomes for SQL analysis; it is not a feature-for-feature replacement.

Next, use the AI quality regression recipe, agent task success and handoff recipe, and accepted output per dollar recipe. For the wider implementation boundary, see AI agent monitoring.