Skip to content
Telemetry
AI and LLM SQL recipe

Find AI quality regressions by prompt version

Compare evaluated quality, acceptance, handoff, and cost by prompt version.

Intermediateai_quality_eventsReviewed 2026-07-28Tested with Apache DataFusion 45.2.0

Reviewed by the Telemetry product team on . We checked the SQL syntax, required event fields, sample results, and limits on using the query. Who reviews this page

Question answered

Did the new prompt version improve quality without increasing human handoffs?

Latency and token cost cannot tell whether an AI workflow is useful. Join prompt versions to reviewed results to compare a rollout with the version it replaced.

Event schema

Fields the query expects

FieldTypeWhy it exists
timestamp_utcTimestampEvaluation completion time in UTC.
workflow_nameUtf8Bounded AI workflow name.
prompt_versionUtf8Reviewed prompt or policy version.
model_nameUtf8Bounded model name.
quality_scoreFloat64Normalized score from a documented evaluator.
quality_passedBooleanWhether the run passed the reviewed quality threshold.
accepted_without_editBooleanWhether a user accepted the output without editing.
human_handoffBooleanWhether the workflow escalated to a person.
cost_usdFloat64Normalized model cost for the run.
environmentUtf8Deployment environment.
DataFusion SQL

Copy the query

sql
SELECT
  workflow_name,
  prompt_version,
  COUNT(*) AS evaluated_runs,
  100.0 * SUM(CASE WHEN quality_passed THEN 1 ELSE 0 END)
    / NULLIF(COUNT(*), 0) AS quality_pass_rate_pct,
  100.0 * SUM(CASE WHEN accepted_without_edit THEN 1 ELSE 0 END)
    / NULLIF(COUNT(*), 0) AS acceptance_rate_pct,
  100.0 * SUM(CASE WHEN human_handoff THEN 1 ELSE 0 END)
    / NULLIF(COUNT(*), 0) AS handoff_rate_pct,
  AVG(quality_score) AS average_quality_score,
  SUM(cost_usd) / NULLIF(
    SUM(CASE WHEN quality_passed THEN 1 ELSE 0 END),
    0
  ) AS cost_per_quality_pass_usd
FROM ai_quality_events
WHERE timestamp_utc >= now() - INTERVAL '30 days'
  AND environment = 'production'
GROUP BY workflow_name, prompt_version
ORDER BY workflow_name, prompt_version;

This read-only query is planned and executed against an empty typed table with Apache DataFusion 45.2.0. We review the synthetic sample output separately. Check field types, thresholds, and counting rules against your own data. Read the testing methodology.

Query result

AI quality by prompt version

Version 4 has the higher quality pass and acceptance rates with no observed handoffs.

workflow_nameprompt_versionevaluated_runsquality_pass_rate_pctacceptance_rate_pcthandoff_rate_pctaverage_quality_scorecost_per_quality_pass_usd
support_replysupport-v356060400.690.03
support_replysupport-v45808000.830.03

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

AI quality by prompt version: static chart of synthetic quality_pass_rate_pct values from the Find AI quality regressions by prompt version example result
Download this SVG chart of the sample results for an article, runbook, or design review. Please credit Telemetry.

Reproduce the example

Download the sample data

The JSON bundle includes the event schema with field types, reproducible input rows, exact SQL, expected output, review notes, and engine version. The CSV contains the displayed result.

How the SQL works

  1. 1Prompt version is an explicit event field, making rollout comparisons reproducible.
  2. 2Quality pass, product acceptance, and handoff rate show different aspects of usefulness.
  3. 3Cost is divided by successful quality outcomes rather than raw runs, which prevents cheap failures from looking efficient.

Edge cases to check

  • Keep evaluator definitions and thresholds stable across the compared versions.
  • Separate offline evaluator scores from real user acceptance instead of treating either as ground truth.
  • Segment by task difficulty or customer cohort when prompt versions receive materially different traffic.

Recommended dashboard

  • Bars: quality and acceptance rate by prompt_version
  • Trend: handoff rate with rollout annotations
  • Table: evaluated runs and cost per quality pass

Alert guidance

Alert on a sustained quality or acceptance regression only after the new version reaches a reviewed evaluation volume.

Read alert setup

Set up the events this query needs

Related instrumentation and guides

Continue the analysis

Run it on your events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key