RAG evaluation with SQL

Evaluate retrieval and answer quality separately. Did the system retrieve relevant evidence? Did the answer stay grounded in that evidence? A single score hides which stage changed.

Version the evaluation contract

Record one terminal event per test case with pipeline_version, test_case_id, relevant_document_retrieved, answer_grounded, retrieval_latency_ms, cost_usd, grader version, and environment. The pipeline version should identify the retriever, index snapshot, reranker, and answer prompt closely enough to reproduce a comparison.

Use the same versioned test set across pipelines. Document the grading rubric and review disagreements between human and model-based graders. Do not put private source passages, raw user questions, complete model answers, secrets, or unrestricted tool output in general telemetry.

For the linked SQL recipe, give each comparison an evaluation_run_id and freeze dataset_version and evaluator_version. Use one terminal record per pipeline_version and test_case_id within that scope. Replace the recipe’s example filters with your actual comparison identifiers. Conflicting records are reported and excluded; null reviews remain unknown. The query counts observed cases, so check the planned case list separately before claiming complete coverage.

Compare quality and constraints

The RAG quality SQL recipe calculates retrieval-relevance rate, grounded-answer rate, average retrieval latency, and evaluation cost for every pipeline version. Its deterministic fixture shows a version that improves both reviewed quality rates while changing cost.

Interpret the columns separately:

1. Poor retrieval relevance points toward indexing, chunking, filtering, embedding, or ranking.
2. Relevant retrieval with poor grounding points toward context construction, prompting, or answer evaluation.
3. Better quality with unacceptable latency or cost may not be ready for production.
4. A small test count should block a release conclusion, not be hidden behind a percentage.

Run the fixture in the browser SQL playground, then replace its test cases with a versioned evaluation set.

Use a release gate

RAG evaluation normally belongs in a release review rather than an on-call page. Set minimum test count, regression tolerances, grader version, and required slices before comparing versions. Keep production feedback as a separate signal with its own consent and privacy rules.

The AI quality evaluation use case covers human handoffs, accepted outcomes, and prompt versions. The AI and LLM recipe collection adds model cost, latency, cache, tool-call, and agent-run analysis.