Measure LLM Cache Savings and Retry Cost

Event contract

Fields the query expects

Field	Type	Why it exists
timestamp_utc	Timestamp	Model request completion time.
model	Utf8	Model identifier.
cache_outcome	Utf8	hit, miss, or unavailable.
attempt	Int64	One-based attempt number.
estimated_cost_usd	Float64	Estimated request cost.

DataFusion SQL

Copy the query

sql

SELECT
  model,
  COUNT(*) AS requests,
  SUM(CASE WHEN cache_outcome = 'hit' THEN 1 ELSE 0 END) AS cache_hits,
  SUM(CASE WHEN attempt > 1 THEN 1 ELSE 0 END) AS retries,
  100.0 * SUM(CASE WHEN cache_outcome = 'hit' THEN 1 ELSE 0 END)
    / NULLIF(COUNT(*), 0) AS cache_hit_rate_pct,
  SUM(CASE WHEN attempt > 1 THEN estimated_cost_usd ELSE 0 END) AS retry_cost_usd,
  SUM(estimated_cost_usd) AS total_cost_usd
FROM llm_requests
WHERE timestamp_utc >= now() - INTERVAL '30 days'
GROUP BY model
ORDER BY retry_cost_usd DESC;

This recipe is read-only and reviewed against Telemetry's DataFusion query patterns. Sample output is deterministic and synthetic; validate field types, thresholds, and business definitions against your own data.

Query result

Retry cost within total model cost

The larger model has both the lower cache-hit rate and the larger retry-cost burden.

gpt-5.1

92.4 USD

1,480.2 USD

gpt-5.1-mini

11.8 USD

604.6 USD

model	requests	cache_hits	retries	cache_hit_rate_pct	retry_cost_usd	total_cost_usd
gpt-5.1	18,400	6,620	740	35.98	92.4	1,480.2
gpt-5.1-mini	49,200	23,810	380	48.39	11.8	604.6

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

How the SQL works

1Cache hit rate counts explicit cache hits within all recorded requests.
2Attempt greater than one isolates repeated model work from first-attempt cost.
3The result explains cost mechanics; connect it to accepted or retained outcomes before optimizing purely for spend.

Edge cases to decide

A provider-side prompt cache and an application response cache are different mechanisms.
Avoid double-counting cost when one logical request emits both attempt and summary events.
Cost estimates must use the price and token rules active when the request ran.