Telemetry
Background jobs SQL recipe

Measure Dead-Letter Queue Growth

Compare dead-letter creation and resolution to find queues accumulating unrecoverable work.

Intermediatejob_eventsReviewed 2026-07-27Tested with Apache DataFusion 45.2.0

Question answered

Which queues are adding dead-letter jobs faster than they are resolving them?

A dead-letter count is a stock, while created and resolved events are flows. Tracking both explains whether the queue is recovering or growing.

Event contract

Fields the query expects

FieldTypeWhy it exists
timestamp_utcTimestampJob lifecycle event time.
job_idUtf8Stable logical job identifier.
queue_nameUtf8Queue that owns the job.
event_nameUtf8dead_letter_created or dead_letter_resolved.
error_typeUtf8Categorized terminal failure.
DataFusion SQL

Copy the query

sql
SELECT
  date_trunc('day', timestamp_utc) AS day,
  queue_name,
  SUM(CASE
    WHEN event_name = 'dead_letter_created' THEN 1 ELSE 0
  END) AS created_jobs,
  SUM(CASE
    WHEN event_name = 'dead_letter_resolved' THEN 1 ELSE 0
  END) AS resolved_jobs,
  SUM(CASE
    WHEN event_name = 'dead_letter_created' THEN 1
    WHEN event_name = 'dead_letter_resolved' THEN -1
    ELSE 0
  END) AS net_queue_change
FROM job_events
WHERE timestamp_utc >= now() - INTERVAL '30 days'
  AND event_name IN ('dead_letter_created', 'dead_letter_resolved')
GROUP BY date_trunc('day', timestamp_utc), queue_name
ORDER BY day, queue_name;

This read-only query is planned and executed against an empty typed table with Apache DataFusion 45.2.0. The deterministic sample output is synthetic and reviewed separately; validate field types, thresholds, and business definitions against your own data. Read the testing methodology.

Query result

Dead-letter jobs created and resolved

Billing accumulated 22 additional dead-letter jobs on July 26 while imports reduced its queue.

dayqueue_namecreated_jobsresolved_jobsnet_queue_change
2026-07-25billing18162
2026-07-26billing31922
2026-07-27imports1214-2

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

How the SQL works

  1. 1Created and resolved events remain separate so the same net change cannot hide very different operational workloads.
  2. 2The signed net change exposes days when unresolved work accumulated.
  3. 3Grouping by queue keeps ownership and escalation paths clear.

Edge cases to decide

  • A retry is not a resolution until the logical job reaches a documented terminal success or discard outcome.
  • Backfilled lifecycle events can change historical net movement.
  • Track the current queue stock separately or calculate a cumulative sum from a known opening balance.

Recommended dashboard

  • Stacked bars: created_jobs and resolved_jobs by queue
  • Trend: cumulative dead-letter stock
  • Table: oldest unresolved jobs by error_type

Alert guidance

Alert when net_queue_change remains positive for consecutive buckets or the oldest unresolved job breaches its response target.

Read alert setup

Put the recipe to work

Related instrumentation and guides

Continue the analysis

Run it on real events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key