Skip to content
Telemetry
AI and LLM SQL recipe

Evaluate RAG retrieval quality by version

Compare relevant-document retrieval, grounded answers, retrieval latency, and evaluation cost across RAG pipeline versions.

Intermediaterag_evaluation_eventsReviewed 2026-07-28Tested with Apache DataFusion 45.2.0

Reviewed by the Telemetry product team on . We checked the SQL syntax, required event fields, sample results, and limits on using the query. Who reviews this page

Question answered

Did the new RAG pipeline improve retrieval and grounded-answer rates?

Compare retrieval relevance, answer grounding, latency, and evaluation cost for the same versioned test set. Check whether a quality gain also costs more or takes longer.

Event schema

Fields the query expects

FieldTypeWhy it exists
timestamp_utcTimestampEvaluation completion time in UTC.
pipeline_versionUtf8Versioned retriever, index, and prompt configuration.
test_case_idUtf8Stable evaluation-case identifier.
relevant_document_retrievedBooleanReviewed retrieval relevance outcome.
answer_groundedBooleanReviewed grounded-answer outcome.
retrieval_latency_msInt64Retrieval-stage latency in milliseconds.
cost_usdFloat64Versioned estimated evaluation cost in USD.
environmentUtf8Evaluation or production environment.
DataFusion SQL

Copy the query

sql
SELECT
  pipeline_version,
  COUNT(*) AS test_cases,
  100.0 * SUM(CASE WHEN relevant_document_retrieved THEN 1 ELSE 0 END)
    / NULLIF(COUNT(*), 0) AS retrieval_relevance_pct,
  100.0 * SUM(CASE WHEN answer_grounded THEN 1 ELSE 0 END)
    / NULLIF(COUNT(*), 0) AS grounded_answer_pct,
  AVG(retrieval_latency_ms) AS avg_retrieval_latency_ms,
  SUM(cost_usd) AS evaluation_cost_usd
FROM rag_evaluation_events
WHERE timestamp_utc >= now() - INTERVAL '30 days'
  AND environment = 'evaluation'
GROUP BY pipeline_version
ORDER BY grounded_answer_pct DESC, retrieval_relevance_pct DESC, pipeline_version;

This read-only query is planned and executed against an empty typed table with Apache DataFusion 45.2.0. We review the synthetic sample output separately. Check field types, thresholds, and counting rules against your own data. Read the testing methodology.

Query result

RAG retrieval and grounding outcomes

Version two improves both reviewed rates in the synthetic evaluation set while costing slightly more.

pipeline_versiontest_casesretrieval_relevance_pctgrounded_answer_pctavg_retrieval_latency_msevaluation_cost_usd
rag-v247575950.05
rag-v1450501200.04

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

RAG retrieval and grounding outcomes: static chart of synthetic retrieval_relevance_pct values from the Evaluate RAG retrieval quality by version example result
Download this SVG chart of the sample results for an article, runbook, or design review. Please credit Telemetry.

Reproduce the example

Download the sample data

The JSON bundle includes the event schema with field types, reproducible input rows, exact SQL, expected output, review notes, and engine version. The CSV contains the displayed result.

Run in browserSample JSONExpected output CSVStatic SVG chart

How the SQL works

  1. 1Pipeline version should identify the retriever, index snapshot, reranker, and answer prompt used for the test.
  2. 2Retrieval relevance and answer grounding remain separate because a good document can still produce an unsupported answer.
  3. 3Latency and cost keep the quality comparison connected to production constraints.

Edge cases to check

  • Use the same versioned test cases across pipelines and report newly added or removed cases.
  • Human and model-based graders need documented rubrics, calibration, and disagreement review.
  • Do not store private source passages or raw answers in general telemetry by default.

Recommended dashboard

  • Stacked bars: retrieval_relevance_pct and grounded_answer_pct by version
  • Trend: evaluation outcomes as the test set changes
  • Table: latency, cost, test count, and grader version

Alert guidance

Use an evaluation release gate with minimum test count and reviewed regression tolerance rather than a production paging alert.

Read alert setup

Set up the events this query needs

Related instrumentation and guides

RAG evaluation SQL guideAI quality use caseAI and LLM recipes

Define the source data

Event schemas for this analysis

rag_retrieval_evaluated

A versioned RAG evaluation event that separates retrieval scores, answer review, latency, and cost without storing source text, prompts, or generated answers.

Review schema

Continue the analysis

Calculate LLM cost by feature and model

Attribute model spend, tokens, request volume, and cost per request to product features.

Open recipe

Measure accepted AI outputs per dollar

Connect model spend to accepted, saved, or otherwise useful product outcomes.

Open recipe

Measure LLM cache savings and retry cost

Compare cached requests, retry volume, and estimated spend by model to find avoidable AI cost.

Open recipe

Run it on your events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key