Telemetry
Structured events SQL recipe

Detect Error-Rate Spikes With a Rolling Baseline

Compare each hourly API error rate with a rolling seven-bucket average instead of relying on one permanent threshold.

Advancedapi_requestsReviewed 2026-07-27

Question answered

Which hourly error-rate buckets are far above their recent baseline?

A fixed threshold misses services with different normal levels. A rolling baseline makes relative changes visible while retaining the raw rate and volume.

Event contract

Fields the query expects

FieldTypeWhy it exists
timestamp_utcTimestampRequest completion time.
status_codeInt64HTTP response status code.
DataFusion SQL

Copy the query

sql
WITH hourly AS (
  SELECT
    date_trunc('hour', timestamp_utc) AS hour,
    COUNT(*) AS requests,
    100.0 * SUM(CASE WHEN status_code >= 500 THEN 1 ELSE 0 END)
      / NULLIF(COUNT(*), 0) AS error_rate_pct
  FROM api_requests
  WHERE timestamp_utc >= now() - INTERVAL '72 hours'
  GROUP BY date_trunc('hour', timestamp_utc)
),
baseline AS (
  SELECT
    hour,
    requests,
    error_rate_pct,
    AVG(error_rate_pct) OVER (
      ORDER BY hour ROWS BETWEEN 7 PRECEDING AND 1 PRECEDING
    ) AS previous_seven_avg_pct
  FROM hourly
)
SELECT
  hour,
  requests,
  error_rate_pct,
  previous_seven_avg_pct,
  error_rate_pct - previous_seven_avg_pct AS increase_pct_points
FROM baseline
WHERE previous_seven_avg_pct IS NOT NULL
ORDER BY hour;

This recipe is read-only and reviewed against Telemetry's DataFusion query patterns. Sample output is deterministic and synthetic; validate field types, thresholds, and business definitions against your own data.

Query result

Hourly error-rate increase above baseline

The 13:00 bucket is the only clear positive deviation from the recent rolling average.

hourrequestserror_rate_pctprevious_seven_avg_pctincrease_pct_points
12:0018,4200.310.280.03
13:0019,1101.420.31.12
14:0018,7800.480.460.02

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

How the SQL works

  1. 1The hourly CTE computes a ratio with its denominator.
  2. 2The window ends at one preceding so the current spike cannot raise its own baseline.
  3. 3The final difference is percentage points, not percent change.

Edge cases to decide

  • A seven-hour baseline does not model daily seasonality.
  • Missing time buckets remain missing; do not silently treat missing traffic as zero errors.
  • Require minimum request volume and combine relative change with an absolute impact threshold.

Recommended dashboard

  • Trend: observed and rolling error rate
  • Bars: increase above baseline
  • Table: routes in anomalous buckets

Alert guidance

Alert only when the increase and absolute error rate both exceed thresholds at meaningful volume.

Read alert setup

Put the recipe to work

Related instrumentation and guides

Continue the analysis

Run it on real events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key