Event contract
Fields the query expects
| Field | Type | Why it exists |
|---|---|---|
| timestamp_utc | Timestamp | Run completion time. |
| job_name | Utf8 | Stable logical job name. |
| queue_wait_ms | Float64 | Milliseconds from scheduled to started. |
| duration_ms | Float64 | Execution duration after start. |
Copy the query
SELECT
job_name,
COUNT(*) AS runs,
approx_percentile_cont(queue_wait_ms, 0.50) AS p50_wait_ms,
approx_percentile_cont(queue_wait_ms, 0.95) AS p95_wait_ms,
approx_percentile_cont(duration_ms, 0.95) AS p95_run_ms
FROM job_runs
WHERE timestamp_utc >= now() - INTERVAL '24 hours'
GROUP BY job_name
HAVING COUNT(*) >= 20
ORDER BY p95_wait_ms DESC;This read-only query is planned and executed against an empty typed table with Apache DataFusion 45.2.0. The deterministic sample output is synthetic and reviewed separately; validate field types, thresholds, and business definitions against your own data. Read the testing methodology.
Query result
p95 queue wait by job
Report generation spends roughly twice as long waiting as running at p95.
| job_name | runs | p50_wait_ms | p95_wait_ms | p95_run_ms |
|---|---|---|---|---|
| generate_report | 340 | 1,800 | 44,200 | 21,800 |
| sync_subscription | 980 | 420 | 7,200 | 8,400 |
| send_email | 12,400 | 80 | 460 | 930 |
Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.
Reproduce the example
Download the public fixture
The JSON bundle includes the typed event contract, illustrative input rows, exact SQL, expected output, review notes, and engine version. The CSV contains the displayed result.
How the SQL works
- 1Queue wait begins when work is scheduled and ends when execution starts.
- 2The p50-to-p95 gap distinguishes a consistently slow queue from intermittent backlog.
- 3Comparing p95 wait with p95 run time shows whether capacity or job code is the larger contributor.
Edge cases to decide
- Use one clock or UTC timestamps for scheduled and started times.
- Scheduled cron delay and queue delay may need separate fields.
- Retries should preserve attempt number so repeated work can be isolated.
Recommended dashboard
- Bars: p50 and p95 wait by job
- Trend: p95 wait by queue
- Table: oldest currently queued items
Alert guidance
Alert when p95 wait exceeds the workflow objective for several complete buckets.
Read alert setupPut the recipe to work
Related instrumentation and guides
Define the source data
Event schemas for this analysis
Continue the analysis
Measure Background Job Retry and Failure Rate
Find unreliable jobs by comparing successful runs, retries, failures, and tail duration.
Open recipeFind Stalled Background Jobs With SQL
Join job start and finish events to identify work that exceeded its expected completion window.
Open recipeMeasure Dead-Letter Queue Growth
Compare dead-letter creation and resolution to find queues accumulating unrecoverable work.
Open recipeRun it on real events
Create a table, adapt the fields, and save the result
Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.