Skip to content
Telemetry
Background jobs SQL recipe

Measure background job retry and failure rate

Find unreliable jobs by comparing successful runs, retries, failures, and tail duration.

Beginnerjob_runsReviewed 2026-07-27Tested with Apache DataFusion 45.2.0

Reviewed by the Telemetry product team on . We checked the SQL syntax, required event fields, sample results, and limits on using the query. Who reviews this page

Question answered

Which background jobs consume the most retries or still fail?

Retries can make a queue appear healthy while hiding extra work and delayed outcomes. This query keeps successful recovery visible without treating it as first-attempt success.

Event schema

Fields the query expects

FieldTypeWhy it exists
timestamp_utcTimestampWhen the run completed or failed.
job_nameUtf8Stable logical job name.
statusUtf8success or failed.
attemptInt64One-based execution attempt.
duration_msFloat64Duration of the attempt.
DataFusion SQL

Copy the query

sql
SELECT
  job_name,
  COUNT(*) AS attempts,
  SUM(CASE WHEN attempt > 1 THEN 1 ELSE 0 END) AS retries,
  SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END) AS failures,
  100.0 * SUM(CASE WHEN attempt > 1 THEN 1 ELSE 0 END)
    / NULLIF(COUNT(*), 0) AS retry_rate_pct,
  approx_percentile_cont(duration_ms, 0.95) AS p95_duration_ms
FROM job_runs
WHERE timestamp_utc >= now() - INTERVAL '7 days'
GROUP BY job_name
ORDER BY retry_rate_pct DESC, failures DESC;

This read-only query is planned and executed against an empty typed table with Apache DataFusion 45.2.0. We review the synthetic sample output separately. Check field types, thresholds, and counting rules against your own data. Read the testing methodology.

Query result

Retry rate by job

Subscription syncs deserve attention even though email has more absolute retry attempts.

job_nameattemptsretriesfailuresretry_rate_pctp95_duration_ms
sync_subscription1032308,000
generate_report81112.512,000
send_email10000900

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

Retry rate by job: static chart of synthetic retry_rate_pct values from the Measure background job retry and failure rate example result
Download this SVG chart of the sample results for an article, runbook, or design review. Please credit Telemetry.

Reproduce the example

Download the sample data

The JSON bundle includes the event schema with field types, reproducible input rows, exact SQL, expected output, review notes, and engine version. The CSV contains the displayed result.

How the SQL works

  1. 1Counting attempts instead of logical jobs intentionally exposes the extra execution load caused by retries.
  2. 2The retry-rate denominator is all attempts. If you prefer a logical-job denominator, include a job_id and count distinct job IDs.
  3. 3p95 duration helps distinguish quick transient retries from expensive runs that also consume worker capacity.

Edge cases to check

  • A scheduled retry may be expected behavior; segment by error_type before changing retry policy.
  • Use a stable job_id if several attempts belong to one logical unit of work.
  • Dead-letter events should be reported separately because they represent exhausted recovery.

Recommended dashboard

  • Bar chart: retry_rate_pct by job_name
  • Line chart: failures by day and job_name
  • Table: latest final failures with customer and error context

Alert guidance

Alert when retry rate doubles relative to the previous seven-day baseline or a final failure affects a critical workflow.

Read alert setup

Set up the events this query needs

Related instrumentation and guides

Continue the analysis

Run it on your events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key