Telemetry
Structured events SQL recipe

Detect Error-Rate Spikes With a Rolling Baseline

Compare each hourly API error rate with a rolling seven-bucket average instead of relying on one permanent threshold.

Advancedapi_requestsReviewed 2026-07-27Tested with Apache DataFusion 45.2.0

Reviewed by the Telemetry product team on . SQL compatibility, event contract, synthetic output, and operational caveats. Review standards and ownership

Question answered

Which hourly error-rate buckets are far above their recent baseline?

A fixed threshold misses services with different normal levels. A rolling baseline makes relative changes visible while retaining the raw rate and volume.

Event contract

Fields the query expects

FieldTypeWhy it exists
timestamp_utcTimestampRequest completion time.
status_codeInt64HTTP response status code.
DataFusion SQL

Copy the query

sql
WITH hourly AS (
  SELECT
    date_trunc('hour', timestamp_utc) AS hour,
    COUNT(*) AS requests,
    100.0 * SUM(CASE WHEN status_code >= 500 THEN 1 ELSE 0 END)
      / NULLIF(COUNT(*), 0) AS error_rate_pct
  FROM api_requests
  WHERE timestamp_utc >= now() - INTERVAL '72 hours'
  GROUP BY date_trunc('hour', timestamp_utc)
),
baseline AS (
  SELECT
    hour,
    requests,
    error_rate_pct,
    AVG(error_rate_pct) OVER (
      ORDER BY hour ROWS BETWEEN 7 PRECEDING AND 1 PRECEDING
    ) AS previous_seven_avg_pct
  FROM hourly
)
SELECT
  hour,
  requests,
  error_rate_pct,
  previous_seven_avg_pct,
  error_rate_pct - previous_seven_avg_pct AS increase_pct_points
FROM baseline
WHERE previous_seven_avg_pct IS NOT NULL
ORDER BY hour;

This read-only query is planned and executed against an empty typed table with Apache DataFusion 45.2.0. The deterministic sample output is synthetic and reviewed separately; validate field types, thresholds, and business definitions against your own data. Read the testing methodology.

Query result

Hourly error-rate increase above baseline

The 13:00 bucket is the only clear positive deviation from the recent rolling average.

hourrequestserror_rate_pctprevious_seven_avg_pctincrease_pct_points
12:0018,4200.310.280.03
13:0019,1101.420.31.12
14:0018,7800.480.460.02

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

Hourly error-rate increase above baseline: static chart of synthetic increase_pct_points values from the Detect Error-Rate Spikes With a Rolling Baseline example result
Indexable SVG of the deterministic example output. Download it for an article, runbook, or design review with attribution.

Reproduce the example

Download the public fixture

The JSON bundle includes the typed event contract, illustrative input rows, exact SQL, expected output, review notes, and engine version. The CSV contains the displayed result.

How the SQL works

  1. 1The hourly CTE computes a ratio with its denominator.
  2. 2The window ends at one preceding so the current spike cannot raise its own baseline.
  3. 3The final difference is percentage points, not percent change.

Edge cases to decide

  • A seven-hour baseline does not model daily seasonality.
  • Missing time buckets remain missing; do not silently treat missing traffic as zero errors.
  • Require minimum request volume and combine relative change with an absolute impact threshold.

Recommended dashboard

  • Trend: observed and rolling error rate
  • Bars: increase above baseline
  • Table: routes in anomalous buckets

Alert guidance

Alert only when the increase and absolute error rate both exceed thresholds at meaningful volume.

Read alert setup

Put the recipe to work

Related instrumentation and guides

Continue the analysis

Run it on real events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key