Event contract
Fields the query expects
| Field | Type | Why it exists |
|---|---|---|
| timestamp_utc | Timestamp | When the event occurred. |
| job_id | Utf8 | Identifier shared by every event for one job. |
| job_name | Utf8 | Stable logical job name. |
| event_name | Utf8 | job_started, job_completed, or job_failed. |
| queue_name | Utf8 | Queue that owns the job. |
Copy the query
WITH started AS (
SELECT
job_id,
job_name,
queue_name,
MIN(timestamp_utc) AS started_at
FROM job_events
WHERE event_name = 'job_started'
AND timestamp_utc >= now() - INTERVAL '24 hours'
GROUP BY job_id, job_name, queue_name
),
finished AS (
SELECT
job_id,
MAX(timestamp_utc) AS finished_at
FROM job_events
WHERE event_name IN ('job_completed', 'job_failed')
AND timestamp_utc >= now() - INTERVAL '24 hours'
GROUP BY job_id
)
SELECT
s.job_id,
s.job_name,
s.queue_name,
s.started_at
FROM started s
LEFT JOIN finished f ON s.job_id = f.job_id
WHERE f.job_id IS NULL
AND s.started_at < now() - INTERVAL '15 minutes'
ORDER BY s.started_at ASC;This recipe is read-only and reviewed against Telemetry's DataFusion query patterns. Sample output is deterministic and synthetic; validate field types, thresholds, and business definitions against your own data.
Query result
Oldest jobs without a terminal event
A table preserves the identifiers an operator needs to inspect or replay the work.
| job_id | job_name | queue_name | started_at |
|---|---|---|---|
| job_7f31 | import_catalog | imports | 2026-07-27 15:02:11Z |
| job_801c | sync_subscription | billing | 2026-07-27 15:19:43Z |
Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.
How the SQL works
- 1The first CTE keeps the earliest start for each logical job. The second keeps the newest terminal event.
- 2A LEFT JOIN preserves started jobs even when no matching completion or failure exists.
- 3The fifteen-minute threshold is an operational definition of stalled. Set it from expected job duration rather than copying it blindly.
Edge cases to decide
- Long-running jobs need job-specific thresholds or a heartbeat event.
- Late events can temporarily create false positives; delay the newest bucket when ingestion may lag.
- A reused job_id will join unrelated runs, so IDs must be unique per logical execution.
Recommended dashboard
- Stat: current stalled-job count
- Table: oldest stalled jobs with job_id and queue_name
- Line chart: stalled-job detections by day
Alert guidance
Alert when any critical job crosses its expected duration plus a short ingestion grace period.
Read alert setupPut the recipe to work
Related instrumentation and guides
Continue the analysis
Run it on real events
Create a table, adapt the fields, and save the result
Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.