Telemetry
AI and LLM SQL recipe

Measure LLM Time to First Token

Compare streaming responsiveness and total generation latency by model and feature.

Intermediatellm_requestsReviewed 2026-07-27Tested with Apache DataFusion 45.2.0

Question answered

Which model and feature combinations feel slow before output begins?

Users experience the pause before streaming separately from total completion time. Measuring both reveals whether latency comes from request startup or generation.

Event contract

Fields the query expects

FieldTypeWhy it exists
timestamp_utcTimestampModel request completion time.
modelUtf8Provider model identifier.
featureUtf8Product feature making the request.
statusUtf8Final request outcome.
time_to_first_token_msFloat64Milliseconds until the first streamed token.
latency_msFloat64Total request duration in milliseconds.
DataFusion SQL

Copy the query

sql
SELECT
  feature,
  model,
  COUNT(*) AS requests,
  approx_percentile_cont(time_to_first_token_ms, 0.50) AS p50_ttft_ms,
  approx_percentile_cont(time_to_first_token_ms, 0.95) AS p95_ttft_ms,
  approx_percentile_cont(latency_ms, 0.95) AS p95_total_latency_ms
FROM llm_requests
WHERE timestamp_utc >= now() - INTERVAL '7 days'
  AND status = 'success'
  AND time_to_first_token_ms IS NOT NULL
GROUP BY feature, model
HAVING COUNT(*) >= 30
ORDER BY p95_ttft_ms DESC;

This read-only query is planned and executed against an empty typed table with Apache DataFusion 45.2.0. The deterministic sample output is synthetic and reviewed separately; validate field types, thresholds, and business definitions against your own data. Read the testing methodology.

Query result

p95 time to first token

Report generation has both the longest startup pause and the slowest total completion.

featuremodelrequestsp50_ttft_msp95_ttft_msp95_total_latency_ms
report_generationlarge-reasoning8421,2804,62018,400
draft_replysmall-fast6,8103109203,840
search_summarybalanced2,4204801,3806,210

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

How the SQL works

  1. 1Successful streaming requests are grouped by the product decision and model.
  2. 2p50 describes the typical pause while p95 exposes the tail users complain about.
  3. 3Total p95 latency remains beside TTFT so optimization does not improve startup while making completion worse.

Edge cases to decide

  • Non-streaming requests do not have a meaningful first-token measurement.
  • Provider-reported timing and client-observed timing can differ; document the measurement boundary.
  • Compare models within features because prompt size and requested output length affect both metrics.

Recommended dashboard

  • Grouped bars: p50_ttft_ms and p95_ttft_ms
  • Trend: p95 TTFT by model and feature
  • Table: slow requests with token counts and retry context

Alert guidance

Alert when a high-volume feature's p95 TTFT exceeds its user-experience objective for consecutive buckets.

Read alert setup

Put the recipe to work

Related instrumentation and guides

Continue the analysis

Run it on real events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key