Event contract
Fields the query expects
| Field | Type | Why it exists |
|---|---|---|
| timestamp_utc | Timestamp | When the workflow outcome was recorded. |
| feature | Utf8 | Stable product feature. |
| model | Utf8 | Provider model identifier. |
| accepted | Boolean | Whether the user accepted or saved the result. |
| estimated_cost_usd | Float64 | Cost of the model request. |
Copy the query
SELECT
feature,
model,
COUNT(*) AS completed_outputs,
SUM(CASE WHEN accepted = true THEN 1 ELSE 0 END) AS accepted_outputs,
100.0 * SUM(CASE WHEN accepted = true THEN 1 ELSE 0 END)
/ NULLIF(COUNT(*), 0) AS acceptance_rate_pct,
SUM(estimated_cost_usd) AS cost_usd,
SUM(CASE WHEN accepted = true THEN 1 ELSE 0 END)
/ NULLIF(SUM(estimated_cost_usd), 0) AS accepted_outputs_per_dollar
FROM llm_requests
WHERE timestamp_utc >= now() - INTERVAL '30 days'
GROUP BY feature, model
HAVING COUNT(*) >= 50
ORDER BY accepted_outputs_per_dollar DESC;This read-only query is planned and executed against an empty typed table with Apache DataFusion 45.2.0. The deterministic sample output is synthetic and reviewed separately; validate field types, thresholds, and business definitions against your own data. Read the testing methodology.
Query result
Accepted outputs per dollar
The largest model wins on acceptance rate but loses on cost-adjusted throughput.
| feature | model | completed_outputs | accepted_outputs | acceptance_rate_pct | cost_usd | accepted_outputs_per_dollar |
|---|---|---|---|---|---|---|
| support_draft | fast-small | 4,200 | 3,108 | 74 | 63 | 49.33 |
| document_summary | balanced-medium | 1,700 | 1,394 | 82 | 78.2 | 17.83 |
| research_report | reasoning-large | 510 | 464 | 90.98 | 215 | 2.16 |
Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.
Reproduce the example
Download the public fixture
The JSON bundle includes the typed event contract, illustrative input rows, exact SQL, expected output, review notes, and engine version. The CSV contains the displayed result.
How the SQL works
- 1Acceptance rate measures quality only among completed outputs. Accepted outputs per dollar adds economic efficiency.
- 2The minimum sample threshold avoids declaring a winner from a handful of requests.
- 3Acceptance must be defined from real product behavior such as saved, sent, applied, or copied—not an assumed model-quality label.
Edge cases to decide
- A discarded output may still be useful if it informed a later action; choose the outcome signal with product context.
- Compare models within the same feature because workflow difficulty changes acceptance.
- Use confidence intervals before making costly model migrations from small differences.
Recommended dashboard
- Bar chart: accepted_outputs_per_dollar by feature and model
- Line chart: acceptance_rate_pct over time
- Table: cost, latency, and outcome for model experiments
Alert guidance
Alert on sharp acceptance-rate drops after a model, prompt, or tool change; review cost efficiency as a dashboard rather than a paging alert.
Read alert setupPut the recipe to work
Related instrumentation and guides
Continue the analysis
Calculate LLM Cost by Feature and Model
Attribute model spend, tokens, request volume, and cost per request to product features.
Open recipeMeasure LLM Cache Savings and Retry Cost
Compare cached requests, retry volume, and estimated spend by model to find avoidable AI cost.
Open recipeMeasure LLM Time to First Token
Compare streaming responsiveness and total generation latency by model and feature.
Open recipeRun it on real events
Create a table, adapt the fields, and save the result
Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.