RAG evaluation with SQL
Evaluate retrieval and answer quality separately. Did the system retrieve relevant evidence? Did the answer stay grounded in that evidence? A single score hides which stage changed.
Version the evaluation contract
Record one terminal event per test case with pipeline_version, test_case_id, relevant_document_retrieved, answer_grounded, retrieval_latency_ms, cost_usd, grader version, and environment. The pipeline version should identify the retriever, index snapshot, reranker, and answer prompt closely enough to reproduce a comparison.
Use the same versioned test set across pipelines. Document the grading rubric and review disagreements between human and model-based graders. Do not put private source passages, raw user questions, complete model answers, secrets, or unrestricted tool output in general telemetry.
Compare quality and constraints
The RAG quality SQL recipe calculates retrieval-relevance rate, grounded-answer rate, average retrieval latency, and evaluation cost for every pipeline version. Its deterministic fixture shows a version that improves both reviewed quality rates while changing cost.
Interpret the columns separately:
- Poor retrieval relevance points toward indexing, chunking, filtering, embedding, or ranking.
- Relevant retrieval with poor grounding points toward context construction, prompting, or answer evaluation.
- Better quality with unacceptable latency or cost may not be ready for production.
- A small test count should block a release conclusion, not be hidden behind a percentage.
Run the fixture in the browser SQL playground, then replace its test cases with a versioned evaluation set.
Use a release gate
RAG evaluation normally belongs in a release review rather than an on-call page. Set minimum test count, regression tolerances, grader version, and required slices before comparing versions. Keep production feedback as a separate signal with its own consent and privacy rules.
The AI quality evaluation use case covers human handoffs, accepted outcomes, and prompt versions. The AI and LLM recipe collection adds model cost, latency, cache, tool-call, and agent-run analysis.