Event contract
Fields the query expects
| Field | Type | Why it exists |
|---|---|---|
| timestamp_utc | Timestamp | When the event occurred. |
| job_id | Utf8 | Identifier shared by every event for one job. |
| job_name | Utf8 | Stable logical job name. |
| event_name | Utf8 | job_started, job_completed, or job_failed. |
| queue_name | Utf8 | Queue that owns the job. |
Copy the query
WITH started AS (
SELECT
job_id,
job_name,
queue_name,
MIN(timestamp_utc) AS started_at
FROM job_events
WHERE event_name = 'job_started'
AND timestamp_utc >= now() - INTERVAL '24 hours'
GROUP BY job_id, job_name, queue_name
),
finished AS (
SELECT
job_id,
MAX(timestamp_utc) AS finished_at
FROM job_events
WHERE event_name IN ('job_completed', 'job_failed')
AND timestamp_utc >= now() - INTERVAL '24 hours'
GROUP BY job_id
)
SELECT
s.job_id,
s.job_name,
s.queue_name,
s.started_at
FROM started s
LEFT JOIN finished f ON s.job_id = f.job_id
WHERE f.job_id IS NULL
AND s.started_at < now() - INTERVAL '15 minutes'
ORDER BY s.started_at ASC;This read-only query is planned and executed against an empty typed table with Apache DataFusion 45.2.0. The deterministic sample output is synthetic and reviewed separately; validate field types, thresholds, and business definitions against your own data. Read the testing methodology.
Query result
Oldest jobs without a terminal event
A table preserves the identifiers an operator needs to inspect or replay the work.
| job_id | job_name | queue_name | started_at |
|---|---|---|---|
| job_7f31 | import_catalog | imports | 2026-07-27 15:02:11Z |
| job_801c | sync_subscription | billing | 2026-07-27 15:19:43Z |
Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.
Reproduce the example
Download the public fixture
The JSON bundle includes the typed event contract, illustrative input rows, exact SQL, expected output, review notes, and engine version. The CSV contains the displayed result.
How the SQL works
- 1The first CTE keeps the earliest start for each logical job. The second keeps the newest terminal event.
- 2A LEFT JOIN preserves started jobs even when no matching completion or failure exists.
- 3The fifteen-minute threshold is an operational definition of stalled. Set it from expected job duration rather than copying it blindly.
Edge cases to decide
- Long-running jobs need job-specific thresholds or a heartbeat event.
- Late events can temporarily create false positives; delay the newest bucket when ingestion may lag.
- A reused job_id will join unrelated runs, so IDs must be unique per logical execution.
Recommended dashboard
- Stat: current stalled-job count
- Table: oldest stalled jobs with job_id and queue_name
- Line chart: stalled-job detections by day
Alert guidance
Alert when any critical job crosses its expected duration plus a short ingestion grace period.
Read alert setupPut the recipe to work
Related instrumentation and guides
Continue the analysis
Measure Background Job Retry and Failure Rate
Find unreliable jobs by comparing successful runs, retries, failures, and tail duration.
Open recipeMeasure Queue Wait Time by Job
Separate time spent waiting in a queue from execution duration and compare p50 and p95 delay by job name.
Open recipeMeasure Dead-Letter Queue Growth
Compare dead-letter creation and resolution to find queues accumulating unrecoverable work.
Open recipeRun it on real events
Create a table, adapt the fields, and save the result
Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.