Common event contract
Fields that keep these queries reusable
- timestamp_utc, run_id, feature, provider, model, and status
- input_tokens, output_tokens, estimated_cost_usd, and latency_ms
- tool_name, retry_count, cache_hit, reviewer_outcome, and human_handoff
Definitions before SQL
Decisions the query cannot make for you
- 1Keep model pricing versioned outside the event producer when possible.
- 2Connect requests to product outcomes without storing raw prompts by default.
- 3Separate provider retries from deliberate user regenerations.
Recommended sequence
Build detection first, then diagnosis
Analysis patterns
Make the result explain a decision
Measure useful output
Connect token, latency, and provider cost to an accepted or completed product outcome instead of optimizing request volume alone.
Preserve the execution path
Keep run, model, tool, cache, and retry fields so expensive or unreliable branches remain visible.
Version changing assumptions
Record model and pricing versions so historical cost and quality comparisons can be reproduced.
Complete recipes
Copy the query, then validate the assumptions
Calculate LLM Cost by Feature and Model
Attribute model spend, tokens, request volume, and cost per request to product features.
Which product features and models are driving LLM spend?
See SQL and resultMeasure Accepted AI Outputs per Dollar
Connect model spend to accepted, saved, or otherwise useful product outcomes.
Which model and feature combination produces the most accepted outputs per dollar?
See SQL and resultMeasure LLM Cache Savings and Retry Cost
Compare cached requests, retry volume, and estimated spend by model to find avoidable AI cost.
How much model cost is associated with retries and cache misses?
See SQL and resultMeasure LLM Time to First Token
Compare streaming responsiveness and total generation latency by model and feature.
Which model and feature combinations feel slow before output begins?
See SQL and resultDetect Repeating AI Agent Tool Loops
Find agent runs that repeatedly call a small set of tools without reaching a successful outcome.
Which agent runs appear stuck in a repetitive tool loop?
See SQL and resultMeasure AI Agent Task Success and Human Handoff
Compare agent task success, reviewer acceptance, cost, latency, and human-handoff rate by workflow and model.
Which agent workflows finish successfully and produce accepted outcomes?
See SQL and resultFind AI Quality Regressions by Prompt Version
Compare evaluated quality, acceptance, handoff, and cost by prompt version.
Did the new prompt version improve quality without increasing human handoffs?
See SQL and resultEvaluate RAG Retrieval Quality by Version
Compare relevant-document retrieval, grounded answers, retrieval latency, and evaluation cost across RAG pipeline versions.
Did the new RAG pipeline improve retrieval and grounded-answer rates?
See SQL and resultAdapt the event contract before the threshold
Keep the analysis pattern, but validate table names, field types, business definitions, time windows, and minimum-volume rules against your own events. Every published query is also planned and executed against an empty typed table with the pinned engine.