Telemetry
Background jobs SQL recipe

Measure Background Job Retry and Failure Rate

Find unreliable jobs by comparing successful runs, retries, failures, and tail duration.

Beginnerjob_runsReviewed 2026-07-27

Question answered

Which background jobs consume the most retries or still fail?

Retries can make a queue appear healthy while hiding extra work and delayed outcomes. This query keeps successful recovery visible without treating it as first-attempt success.

Event contract

Fields the query expects

FieldTypeWhy it exists
timestamp_utcTimestampWhen the run completed or failed.
job_nameUtf8Stable logical job name.
statusUtf8success or failed.
attemptInt64One-based execution attempt.
duration_msFloat64Duration of the attempt.
DataFusion SQL

Copy the query

sql
SELECT
  job_name,
  COUNT(*) AS attempts,
  SUM(CASE WHEN attempt > 1 THEN 1 ELSE 0 END) AS retries,
  SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END) AS failures,
  100.0 * SUM(CASE WHEN attempt > 1 THEN 1 ELSE 0 END)
    / NULLIF(COUNT(*), 0) AS retry_rate_pct,
  approx_percentile_cont(duration_ms, 0.95) AS p95_duration_ms
FROM job_runs
WHERE timestamp_utc >= now() - INTERVAL '7 days'
GROUP BY job_name
ORDER BY retry_rate_pct DESC, failures DESC;

This recipe is read-only and reviewed against Telemetry's DataFusion query patterns. Sample output is deterministic and synthetic; validate field types, thresholds, and business definitions against your own data.

Query result

Retry rate by job

Subscription syncs deserve attention even though email has more absolute retry attempts.

job_nameattemptsretriesfailuresretry_rate_pctp95_duration_ms
sync_subscription9801121811.438,400
generate_report4223177.3522,100
send_email12,4009640.77930

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

How the SQL works

  1. 1Counting attempts instead of logical jobs intentionally exposes the extra execution load caused by retries.
  2. 2The retry-rate denominator is all attempts. If you prefer a logical-job denominator, include a job_id and count distinct job IDs.
  3. 3p95 duration helps distinguish quick transient retries from expensive runs that also consume worker capacity.

Edge cases to decide

  • A scheduled retry may be expected behavior; segment by error_type before changing retry policy.
  • Use a stable job_id if several attempts belong to one logical unit of work.
  • Dead-letter events should be reported separately because they represent exhausted recovery.

Recommended dashboard

  • Bar chart: retry_rate_pct by job_name
  • Line chart: failures by day and job_name
  • Table: latest final failures with customer and error context

Alert guidance

Alert when retry rate doubles relative to the previous seven-day baseline or a final failure affects a critical workflow.

Read alert setup

Put the recipe to work

Related instrumentation and guides

Continue the analysis

Run it on real events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key