Telemetry
Background jobs SQL recipe

Measure Background Job Retry and Failure Rate

Find unreliable jobs by comparing successful runs, retries, failures, and tail duration.

Beginnerjob_runsReviewed 2026-07-27Tested with Apache DataFusion 45.2.0

Reviewed by the Telemetry product team on . SQL compatibility, event contract, synthetic output, and operational caveats. Review standards and ownership

Question answered

Which background jobs consume the most retries or still fail?

Retries can make a queue appear healthy while hiding extra work and delayed outcomes. This query keeps successful recovery visible without treating it as first-attempt success.

Event contract

Fields the query expects

FieldTypeWhy it exists
timestamp_utcTimestampWhen the run completed or failed.
job_nameUtf8Stable logical job name.
statusUtf8success or failed.
attemptInt64One-based execution attempt.
duration_msFloat64Duration of the attempt.
DataFusion SQL

Copy the query

sql
SELECT
  job_name,
  COUNT(*) AS attempts,
  SUM(CASE WHEN attempt > 1 THEN 1 ELSE 0 END) AS retries,
  SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END) AS failures,
  100.0 * SUM(CASE WHEN attempt > 1 THEN 1 ELSE 0 END)
    / NULLIF(COUNT(*), 0) AS retry_rate_pct,
  approx_percentile_cont(duration_ms, 0.95) AS p95_duration_ms
FROM job_runs
WHERE timestamp_utc >= now() - INTERVAL '7 days'
GROUP BY job_name
ORDER BY retry_rate_pct DESC, failures DESC;

This read-only query is planned and executed against an empty typed table with Apache DataFusion 45.2.0. The deterministic sample output is synthetic and reviewed separately; validate field types, thresholds, and business definitions against your own data. Read the testing methodology.

Query result

Retry rate by job

Subscription syncs deserve attention even though email has more absolute retry attempts.

job_nameattemptsretriesfailuresretry_rate_pctp95_duration_ms
sync_subscription1032308,000
generate_report81112.512,000
send_email10000900

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

Retry rate by job: static chart of synthetic retry_rate_pct values from the Measure Background Job Retry and Failure Rate example result
Indexable SVG of the deterministic example output. Download it for an article, runbook, or design review with attribution.

Reproduce the example

Download the public fixture

The JSON bundle includes the typed event contract, reproducible input rows, exact SQL, expected output, review notes, and engine version. The CSV contains the displayed result.

How the SQL works

  1. 1Counting attempts instead of logical jobs intentionally exposes the extra execution load caused by retries.
  2. 2The retry-rate denominator is all attempts. If you prefer a logical-job denominator, include a job_id and count distinct job IDs.
  3. 3p95 duration helps distinguish quick transient retries from expensive runs that also consume worker capacity.

Edge cases to decide

  • A scheduled retry may be expected behavior; segment by error_type before changing retry policy.
  • Use a stable job_id if several attempts belong to one logical unit of work.
  • Dead-letter events should be reported separately because they represent exhausted recovery.

Recommended dashboard

  • Bar chart: retry_rate_pct by job_name
  • Line chart: failures by day and job_name
  • Table: latest final failures with customer and error context

Alert guidance

Alert when retry rate doubles relative to the previous seven-day baseline or a final failure affects a critical workflow.

Read alert setup

Put the recipe to work

Related instrumentation and guides

Continue the analysis

Run it on real events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key