Event schema
Fields the query expects
| Field | Type | Why it exists |
|---|---|---|
| timestamp_utc | Timestamp | When the event occurred. |
| job_id | Utf8 | Identifier shared by every event for one job. |
| job_name | Utf8 | Stable logical job name. |
| event_name | Utf8 | job_started, job_completed, or job_failed. |
| queue_name | Utf8 | Queue that owns the job. |
Copy the query
WITH started AS (
SELECT
job_id,
job_name,
queue_name,
MIN(timestamp_utc) AS started_at
FROM job_events
WHERE event_name = 'job_started'
AND timestamp_utc >= now() - INTERVAL '24 hours'
GROUP BY job_id, job_name, queue_name
),
finished AS (
SELECT
job_id,
MAX(timestamp_utc) AS finished_at
FROM job_events
WHERE event_name IN ('job_completed', 'job_failed')
AND timestamp_utc >= now() - INTERVAL '24 hours'
GROUP BY job_id
)
SELECT
s.job_id,
s.job_name,
s.queue_name,
s.started_at
FROM started s
LEFT JOIN finished f ON s.job_id = f.job_id
WHERE f.job_id IS NULL
AND s.started_at < now() - INTERVAL '15 minutes'
ORDER BY s.started_at ASC;This read-only query is planned and executed against an empty typed table with Apache DataFusion 45.2.0. We review the synthetic sample output separately. Check field types, thresholds, and counting rules against your own data. Read the testing methodology.
Query result
Oldest jobs without a terminal event
A table preserves the identifiers an operator needs to inspect or replay the work.
| job_id | job_name | queue_name | started_at |
|---|---|---|---|
| job_7f31 | import_catalog | imports | 2026-07-27 15:02:11Z |
| job_801c | sync_subscription | billing | 2026-07-27 15:19:43Z |
Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.
Reproduce the example
Download the sample data
The JSON bundle includes the event schema with field types, illustrative input rows, exact SQL, expected output, review notes, and engine version. The CSV contains the displayed result.
How the SQL works
- 1The first CTE keeps the earliest start for each logical job. The second keeps the newest final event.
- 2A LEFT JOIN preserves started jobs even when no matching completion or failure exists.
- 3The fifteen-minute threshold is an operational definition of stalled. Set it from expected job duration rather than copying it blindly.
Edge cases to check
- Long-running jobs need job-specific thresholds or a heartbeat event.
- Late events can temporarily create false positives; delay the newest bucket when ingestion may lag.
- A reused job_id will join unrelated runs, so IDs must be unique per logical execution.
Recommended dashboard
- Stat: current stalled-job count
- Table: oldest stalled jobs with job_id and queue_name
- Line chart: stalled-job detections by day
Alert guidance
Alert when any critical job crosses its expected duration plus a short ingestion grace period.
Read alert setupSet up the events this query needs
Related instrumentation and guides
Continue the analysis
Measure background job retry and failure rate
Find unreliable jobs by comparing successful runs, retries, failures, and tail duration.
Open recipeMeasure queue wait time by job
Separate time spent waiting in a queue from execution duration and compare p50 and p95 delay by job name.
Open recipeMeasure dead-letter queue growth
Compare dead-letter creation and resolution to find queues accumulating unrecoverable work.
Open recipeRun it on your events
Create a table, adapt the fields, and save the result
Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.