Telemetry
AI and LLM SQL recipe

Evaluate RAG Retrieval Quality by Version

Compare relevant-document retrieval, grounded answers, retrieval latency, and evaluation cost across RAG pipeline versions.

Intermediaterag_evaluation_eventsReviewed 2026-07-28Tested with Apache DataFusion 45.2.0

Reviewed by the Telemetry product team on . SQL compatibility, event contract, synthetic output, and operational caveats. Editorial ownership

Question answered

Did the new RAG pipeline improve retrieval and grounded-answer rates?

RAG quality is not one score. This query keeps retrieval relevance, answer grounding, latency, and evaluation cost together for a versioned test set so an apparent quality gain can be reviewed against operational tradeoffs.

Event contract

Fields the query expects

FieldTypeWhy it exists
timestamp_utcTimestampEvaluation completion time in UTC.
pipeline_versionUtf8Versioned retriever, index, and prompt configuration.
test_case_idUtf8Stable evaluation-case identifier.
relevant_document_retrievedBooleanReviewed retrieval relevance outcome.
answer_groundedBooleanReviewed grounded-answer outcome.
retrieval_latency_msInt64Retrieval-stage latency in milliseconds.
cost_usdFloat64Versioned estimated evaluation cost in USD.
environmentUtf8Evaluation or production environment.
DataFusion SQL

Copy the query

sql
SELECT
  pipeline_version,
  COUNT(*) AS test_cases,
  100.0 * SUM(CASE WHEN relevant_document_retrieved THEN 1 ELSE 0 END)
    / NULLIF(COUNT(*), 0) AS retrieval_relevance_pct,
  100.0 * SUM(CASE WHEN answer_grounded THEN 1 ELSE 0 END)
    / NULLIF(COUNT(*), 0) AS grounded_answer_pct,
  AVG(retrieval_latency_ms) AS avg_retrieval_latency_ms,
  SUM(cost_usd) AS evaluation_cost_usd
FROM rag_evaluation_events
WHERE timestamp_utc >= now() - INTERVAL '30 days'
  AND environment = 'evaluation'
GROUP BY pipeline_version
ORDER BY grounded_answer_pct DESC, retrieval_relevance_pct DESC, pipeline_version;

This read-only query is planned and executed against an empty typed table with Apache DataFusion 45.2.0. The deterministic sample output is synthetic and reviewed separately; validate field types, thresholds, and business definitions against your own data. Read the testing methodology.

Query result

RAG retrieval and grounding outcomes

Version two improves both reviewed rates in the synthetic evaluation set while costing slightly more.

pipeline_versiontest_casesretrieval_relevance_pctgrounded_answer_pctavg_retrieval_latency_msevaluation_cost_usd
rag-v247575950.05
rag-v1450501200.04

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

RAG retrieval and grounding outcomes: static chart of synthetic retrieval_relevance_pct values from the Evaluate RAG Retrieval Quality by Version example result
Indexable SVG of the deterministic example output. Download it for an article, runbook, or design review with attribution.

Reproduce the example

Download the public fixture

The JSON bundle includes the typed event contract, reproducible input rows, exact SQL, expected output, review notes, and engine version. The CSV contains the displayed result.

Run in browserJSON fixtureExpected output CSVStatic SVG chart

How the SQL works

  1. 1Pipeline version should identify the retriever, index snapshot, reranker, and answer prompt used for the test.
  2. 2Retrieval relevance and answer grounding remain separate because a good document can still produce an unsupported answer.
  3. 3Latency and cost keep the quality comparison connected to production constraints.

Edge cases to decide

  • Use the same versioned test cases across pipelines and report newly added or removed cases.
  • Human and model-based graders need documented rubrics, calibration, and disagreement review.
  • Do not store private source passages or raw answers in general telemetry by default.

Recommended dashboard

  • Stacked bars: retrieval_relevance_pct and grounded_answer_pct by version
  • Trend: evaluation outcomes as the test set changes
  • Table: latency, cost, test count, and grader version

Alert guidance

Use an evaluation release gate with minimum test count and reviewed regression tolerance rather than a production paging alert.

Read alert setup

Put the recipe to work

Related instrumentation and guides

RAG evaluation SQL guideAI quality use caseAI and LLM recipes

Continue the analysis

Calculate LLM Cost by Feature and Model

Attribute model spend, tokens, request volume, and cost per request to product features.

Open recipe

Measure Accepted AI Outputs per Dollar

Connect model spend to accepted, saved, or otherwise useful product outcomes.

Open recipe

Measure LLM Cache Savings and Retry Cost

Compare cached requests, retry volume, and estimated spend by model to find avoidable AI cost.

Open recipe

Run it on real events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key