Telemetry
Webhooks SQL recipe

Measure Webhook Retry Recovery

Separate permanent webhook failures from deliveries that recovered on a later attempt.

Intermediatewebhook_deliveriesReviewed 2026-07-27

Question answered

Are webhook retries recovering failures or creating more work?

A raw failure count overstates customer impact when providers retry successfully. This query shows initial failures, recovered deliveries, permanent failures, and duplicate suppression together.

Event contract

Fields the query expects

FieldTypeWhy it exists
timestamp_utcTimestampWhen the processing attempt ended.
delivery_idUtf8Stable identifier shared by retry attempts.
providerUtf8Webhook provider.
event_typeUtf8Provider event type.
attemptInt64One-based processing attempt.
statusUtf8success, failed, or deduplicated.
DataFusion SQL

Copy the query

sql
WITH delivery_outcomes AS (
  SELECT
    delivery_id,
    provider,
    event_type,
    MAX(attempt) AS attempts,
    SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END) AS failed_attempts,
    SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END) AS successful_attempts,
    SUM(CASE WHEN status = 'deduplicated' THEN 1 ELSE 0 END) AS deduplicated_attempts
  FROM webhook_deliveries
  WHERE timestamp_utc >= now() - INTERVAL '7 days'
  GROUP BY delivery_id, provider, event_type
)
SELECT
  provider,
  event_type,
  COUNT(*) AS deliveries,
  SUM(CASE
    WHEN failed_attempts > 0 AND successful_attempts > 0 THEN 1 ELSE 0
  END) AS recovered,
  SUM(CASE
    WHEN failed_attempts > 0 AND successful_attempts = 0 THEN 1 ELSE 0
  END) AS permanent_failures,
  SUM(deduplicated_attempts) AS duplicates_suppressed
FROM delivery_outcomes
GROUP BY provider, event_type
ORDER BY permanent_failures DESC, recovered DESC;

This recipe is read-only and reviewed against Telemetry's DataFusion query patterns. Sample output is deterministic and synthetic; validate field types, thresholds, and business definitions against your own data.

Query result

Permanent failures by webhook event

Recovered events remain visible without being counted as permanent customer impact.

providerevent_typedeliveriesrecoveredpermanent_failuresduplicates_suppressed
stripeinvoice.payment_succeeded2,18041673
githubpush6,42018229

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

How the SQL works

  1. 1The first CTE collapses all attempts for one delivery into a single outcome record.
  2. 2A recovered delivery has at least one failed attempt and at least one successful attempt. A permanent failure never recorded success.
  3. 3Duplicate suppression is tracked separately because correct idempotency is a healthy outcome, not a processing failure.

Edge cases to decide

  • Retain provider delivery IDs so retries can be grouped reliably.
  • A delivery may recover after the query window; use a window long enough to cover the provider's retry schedule.
  • Separate receipt from downstream side effects when success requires more than acknowledging the provider.

Recommended dashboard

  • Grouped bars: recovered and permanent_failures by event_type
  • Stat: duplicate suppression rate
  • Table: latest permanent failures with downstream job context

Alert guidance

Alert on permanent failures immediately for revenue-critical events, and on a sudden increase in recovered deliveries as an early-warning signal.

Read alert setup

Put the recipe to work

Related instrumentation and guides

Continue the analysis

Run it on real events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key