Telemetry
Dashboard example

Background job reliability dashboard

Separate terminal job outcomes from attempts, then compare retries, queue wait, and duration by stable job name.

Reviewed by the Telemetry product team on . Decision, event grain, metrics, DataFusion SQL, synthetic result, assumptions, freshness, and interpretation boundary. Review standards and ownership

Row grain

One terminal outcome per logical job.

Decision supported

Choose whether to inspect capacity, retry policy, job code, or a dependency.

Metric contract

Put the denominator and units beside the chart

Adapt these names to your contract while preserving the declared grain. Inspect the linked event schema before mapping fields from production producers.

  • Completed jobs
  • Terminal failure rate
  • Retry rate
  • p95 duration
Inspect the event schema
Complete DataFusion SQL

Review the query before adapting the fields

This public query is a starting definition. Add a bounded time filter, complete-bucket policy, environment, and minimum volume appropriate to your production event contract.

SELECT
  job_name,
  COUNT(*) AS completed_jobs,
  ROUND(
    100.0 * SUM(CASE WHEN status = 'error' THEN 1 ELSE 0 END)
    / NULLIF(COUNT(*), 0),
    2
  ) AS terminal_failure_rate_pct,
  ROUND(AVG(attempt_count), 2) AS average_attempts,
  approx_percentile_cont(duration_ms, 0.95) AS p95_duration_ms
FROM job_runs
GROUP BY job_name
ORDER BY terminal_failure_rate_pct DESC, completed_jobs DESC;
Data assumptions

Prove these before publishing the result

  • A stable job identifier connects attempts without exposing job payloads.
  • Attempt count includes the terminal attempt.
  • Queue wait and execution duration use separate millisecond fields.
Query review

Checks that keep the dashboard trustworthy

  • Use attempt-level data only when the denominator is explicitly attempts.
  • Add queue wait percentiles beside execution duration.
  • Use start and terminal lifecycle events to detect vanished workers.

Interpretation boundary

What this result cannot prove by itself

Count one terminal outcome per job. Attempt-level logs inflate volume and can make a recovered retry look like a failed customer workflow.

Related dashboard examples

Validate the definition with known data

Run a fixture with known success, failure, missing, duplicate, and boundary cases before connecting the query to a production dashboard or alert.