Telemetry
Infrastructure SQL recipe

Calculate Incident Detection and Recovery Time

Calculate time to detect and time to recover from structured incident lifecycle events.

Advancedincident_lifecycle_eventsReviewed 2026-07-28Tested with Apache DataFusion 45.2.0

Reviewed by the Telemetry product team on . SQL compatibility, event contract, synthetic output, and operational caveats. Editorial ownership

Question answered

How long does each service take to detect and recover from incidents?

An incident timeline should separate customer impact, detection, and resolution. This query pivots lifecycle events into one record per incident before calculating service-level detection and recovery averages.

Event contract

Fields the query expects

FieldTypeWhy it exists
timestamp_utcTimestampLifecycle event time in UTC.
incident_idUtf8Stable incident identifier.
serviceUtf8Primary affected service.
event_nameUtf8impact_started, detected, or resolved.
severityUtf8Reviewed incident severity.
environmentUtf8Deployment environment.
DataFusion SQL

Copy the query

sql
WITH incident_times AS (
  SELECT
    incident_id,
    service,
    MIN(CASE WHEN event_name = 'impact_started' THEN timestamp_utc END)
      AS impact_started_at,
    MIN(CASE WHEN event_name = 'detected' THEN timestamp_utc END)
      AS detected_at,
    MAX(CASE WHEN event_name = 'resolved' THEN timestamp_utc END)
      AS resolved_at
  FROM incident_lifecycle_events
  WHERE timestamp_utc >= now() - INTERVAL '90 days'
    AND environment = 'production'
  GROUP BY incident_id, service
)
SELECT
  service,
  COUNT(*) AS incidents,
  AVG(date_part('epoch', detected_at - impact_started_at) / 60.0)
    AS average_detection_minutes,
  AVG(date_part('epoch', resolved_at - detected_at) / 60.0)
    AS average_recovery_minutes
FROM incident_times
WHERE impact_started_at IS NOT NULL
  AND detected_at IS NOT NULL
  AND resolved_at IS NOT NULL
GROUP BY service
ORDER BY average_recovery_minutes DESC, service;

This read-only query is planned and executed against an empty typed table with Apache DataFusion 45.2.0. The deterministic sample output is synthetic and reviewed separately; validate field types, thresholds, and business definitions against your own data. Read the testing methodology.

Query result

Incident detection and recovery time

Billing-worker has the longer observed detection and recovery intervals.

serviceincidentsaverage_detection_minutesaverage_recovery_minutes
billing-worker11560
checkout-api27.532.5

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

Incident detection and recovery time: static chart of synthetic average_recovery_minutes values from the Calculate Incident Detection and Recovery Time example result
Indexable SVG of the deterministic example output. Download it for an article, runbook, or design review with attribution.

Reproduce the example

Download the public fixture

The JSON bundle includes the typed event contract, reproducible input rows, exact SQL, expected output, review notes, and engine version. The CSV contains the displayed result.

How the SQL works

  1. 1Lifecycle events are pivoted to one row per incident before duration calculations.
  2. 2Detection measures impact start to detection; recovery measures detection to resolution.
  3. 3Incomplete incidents are excluded from the completed-duration average and should be reported separately.

Edge cases to decide

  • Define whether recovery ends at mitigation, full restoration, or incident closure and use one rule consistently.
  • Reopened incidents may need multiple impact intervals rather than one maximum resolved time.
  • Averages are illustrative; report medians and percentiles when incident volume is sufficient.

Recommended dashboard

  • Bars: average detection and recovery minutes by service
  • Trend: completed incident durations by month
  • Table: open incidents missing a resolved event

Alert guidance

Use operational alerts for live impact; use this result as a response-quality review and trend metric.

Read alert setup

Put the recipe to work

Related instrumentation and guides

Continue the analysis

Run it on real events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key