Telemetry
Background jobs SQL recipe

Measure Queue Wait Time by Job

Separate time spent waiting in a queue from execution duration and compare p50 and p95 delay by job name.

Beginnerjob_runsReviewed 2026-07-27

Question answered

Which jobs wait longest before a worker starts them?

A fast worker can still produce a slow customer outcome when work sits in the queue. Queue wait and execution duration need separate fields and objectives.

Event contract

Fields the query expects

FieldTypeWhy it exists
timestamp_utcTimestampRun completion time.
job_nameUtf8Stable logical job name.
queue_wait_msFloat64Milliseconds from scheduled to started.
duration_msFloat64Execution duration after start.
DataFusion SQL

Copy the query

sql
SELECT
  job_name,
  COUNT(*) AS runs,
  approx_percentile_cont(queue_wait_ms, 0.50) AS p50_wait_ms,
  approx_percentile_cont(queue_wait_ms, 0.95) AS p95_wait_ms,
  approx_percentile_cont(duration_ms, 0.95) AS p95_run_ms
FROM job_runs
WHERE timestamp_utc >= now() - INTERVAL '24 hours'
GROUP BY job_name
HAVING COUNT(*) >= 20
ORDER BY p95_wait_ms DESC;

This recipe is read-only and reviewed against Telemetry's DataFusion query patterns. Sample output is deterministic and synthetic; validate field types, thresholds, and business definitions against your own data.

Query result

p95 queue wait by job

Report generation spends roughly twice as long waiting as running at p95.

job_namerunsp50_wait_msp95_wait_msp95_run_ms
generate_report3401,80044,20021,800
sync_subscription9804207,2008,400
send_email12,40080460930

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

How the SQL works

  1. 1Queue wait begins when work is scheduled and ends when execution starts.
  2. 2The p50-to-p95 gap distinguishes a consistently slow queue from intermittent backlog.
  3. 3Comparing p95 wait with p95 run time shows whether capacity or job code is the larger contributor.

Edge cases to decide

  • Use one clock or UTC timestamps for scheduled and started times.
  • Scheduled cron delay and queue delay may need separate fields.
  • Retries should preserve attempt number so repeated work can be isolated.

Recommended dashboard

  • Bars: p50 and p95 wait by job
  • Trend: p95 wait by queue
  • Table: oldest currently queued items

Alert guidance

Alert when p95 wait exceeds the workflow objective for several complete buckets.

Read alert setup

Put the recipe to work

Related instrumentation and guides

Queue worker guideBackground job use caseStalled job recipe

Continue the analysis

Measure Background Job Retry and Failure Rate

Find unreliable jobs by comparing successful runs, retries, failures, and tail duration.

Open recipe

Find Stalled Background Jobs With SQL

Join job start and finish events to identify work that exceeded its expected completion window.

Open recipe

Run it on real events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key