RAG Evaluation with SQL

Retrieval-augmented generation has at least two distinct quality boundaries: whether the system retrieved relevant evidence and whether the final answer stayed grounded in that evidence. Combining them into one score hides the part of the pipeline that changed.

Version the evaluation contract

Record one terminal event per test case with pipeline_version, test_case_id, relevant_document_retrieved, answer_grounded, retrieval_latency_ms, cost_usd, grader version, and environment. The pipeline version should identify the retriever, index snapshot, reranker, and answer prompt closely enough to reproduce a comparison.

Use the same versioned test set across pipelines. Document the grading rubric and review disagreements between human and model-based graders. Do not put private source passages, raw user questions, complete model answers, secrets, or unrestricted tool output in general telemetry.

Compare quality and constraints

The RAG quality SQL recipe calculates retrieval-relevance rate, grounded-answer rate, average retrieval latency, and evaluation cost for every pipeline version. Its deterministic fixture shows a version that improves both reviewed quality rates while changing cost.

Interpret the columns separately:

1. Poor retrieval relevance points toward indexing, chunking, filtering, embedding, or ranking.
2. Relevant retrieval with poor grounding points toward context construction, prompting, or answer evaluation.
3. Better quality with unacceptable latency or cost may not be ready for production.
4. A small test count should block a release conclusion, not be hidden behind a percentage.

Run the fixture in the browser SQL playground, then replace its test cases with a versioned evaluation set.

Use a release gate

RAG evaluation normally belongs in a release review rather than an on-call page. Set minimum test count, regression tolerances, grader version, and required slices before comparing versions. Keep production feedback as a separate signal with its own consent and privacy rules.

The AI quality evaluation use case covers human handoffs, accepted outcomes, and prompt versions. The AI and LLM recipe collection adds model cost, latency, cache, tool-call, and agent-run analysis.