Telemetry
Background jobs SQL recipe

Find Stalled Background Jobs With SQL

Join job start and finish events to identify work that exceeded its expected completion window.

Intermediatejob_eventsReviewed 2026-07-27

Question answered

Which jobs started but never produced a terminal event?

A job can disappear without producing a failure event. Joining starts to terminal events reveals work that is still running, lost after a worker crash, or missing its completion instrumentation.

Event contract

Fields the query expects

FieldTypeWhy it exists
timestamp_utcTimestampWhen the event occurred.
job_idUtf8Identifier shared by every event for one job.
job_nameUtf8Stable logical job name.
event_nameUtf8job_started, job_completed, or job_failed.
queue_nameUtf8Queue that owns the job.
DataFusion SQL

Copy the query

sql
WITH started AS (
  SELECT
    job_id,
    job_name,
    queue_name,
    MIN(timestamp_utc) AS started_at
  FROM job_events
  WHERE event_name = 'job_started'
    AND timestamp_utc >= now() - INTERVAL '24 hours'
  GROUP BY job_id, job_name, queue_name
),
finished AS (
  SELECT
    job_id,
    MAX(timestamp_utc) AS finished_at
  FROM job_events
  WHERE event_name IN ('job_completed', 'job_failed')
    AND timestamp_utc >= now() - INTERVAL '24 hours'
  GROUP BY job_id
)
SELECT
  s.job_id,
  s.job_name,
  s.queue_name,
  s.started_at
FROM started s
LEFT JOIN finished f ON s.job_id = f.job_id
WHERE f.job_id IS NULL
  AND s.started_at < now() - INTERVAL '15 minutes'
ORDER BY s.started_at ASC;

This recipe is read-only and reviewed against Telemetry's DataFusion query patterns. Sample output is deterministic and synthetic; validate field types, thresholds, and business definitions against your own data.

Query result

Oldest jobs without a terminal event

A table preserves the identifiers an operator needs to inspect or replay the work.

job_idjob_namequeue_namestarted_at
job_7f31import_catalogimports2026-07-27 15:02:11Z
job_801csync_subscriptionbilling2026-07-27 15:19:43Z

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

How the SQL works

  1. 1The first CTE keeps the earliest start for each logical job. The second keeps the newest terminal event.
  2. 2A LEFT JOIN preserves started jobs even when no matching completion or failure exists.
  3. 3The fifteen-minute threshold is an operational definition of stalled. Set it from expected job duration rather than copying it blindly.

Edge cases to decide

  • Long-running jobs need job-specific thresholds or a heartbeat event.
  • Late events can temporarily create false positives; delay the newest bucket when ingestion may lag.
  • A reused job_id will join unrelated runs, so IDs must be unique per logical execution.

Recommended dashboard

  • Stat: current stalled-job count
  • Table: oldest stalled jobs with job_id and queue_name
  • Line chart: stalled-job detections by day

Alert guidance

Alert when any critical job crosses its expected duration plus a short ingestion grace period.

Read alert setup

Put the recipe to work

Related instrumentation and guides

Continue the analysis

Run it on real events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key