Skip to content
Telemetry
Webhooks SQL recipe

Measure webhook retry recovery

Separate permanent webhook failures from deliveries that recovered on a later attempt.

Intermediatewebhook_deliveriesReviewed 2026-07-27Tested with Apache DataFusion 45.2.0

Reviewed by the Telemetry product team on . We checked the SQL syntax, required event fields, sample results, and limits on using the query. Who reviews this page

Question answered

Are webhook retries recovering failures or creating more work?

A raw failure count overstates customer impact when providers retry successfully. This query shows initial failures, recovered deliveries, permanent failures, and duplicate suppression together.

Event schema

Fields the query expects

FieldTypeWhy it exists
timestamp_utcTimestampWhen the processing attempt ended.
delivery_idUtf8Stable identifier shared by retry attempts.
providerUtf8Webhook provider.
event_typeUtf8Provider event type.
attemptInt64One-based processing attempt.
statusUtf8success, failed, or deduplicated.
DataFusion SQL

Copy the query

sql
WITH delivery_outcomes AS (
  SELECT
    delivery_id,
    provider,
    event_type,
    MAX(attempt) AS attempts,
    SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END) AS failed_attempts,
    SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END) AS successful_attempts,
    SUM(CASE WHEN status = 'deduplicated' THEN 1 ELSE 0 END) AS deduplicated_attempts
  FROM webhook_deliveries
  WHERE timestamp_utc >= now() - INTERVAL '7 days'
  GROUP BY delivery_id, provider, event_type
)
SELECT
  provider,
  event_type,
  COUNT(*) AS deliveries,
  SUM(CASE
    WHEN failed_attempts > 0 AND successful_attempts > 0 THEN 1 ELSE 0
  END) AS recovered,
  SUM(CASE
    WHEN failed_attempts > 0 AND successful_attempts = 0 THEN 1 ELSE 0
  END) AS permanent_failures,
  SUM(deduplicated_attempts) AS duplicates_suppressed
FROM delivery_outcomes
GROUP BY provider, event_type
ORDER BY permanent_failures DESC, recovered DESC;

This read-only query is planned and executed against an empty typed table with Apache DataFusion 45.2.0. We review the synthetic sample output separately. Check field types, thresholds, and counting rules against your own data. Read the testing methodology.

Query result

Permanent failures by webhook event

Recovered events remain visible without being counted as permanent customer impact.

providerevent_typedeliveriesrecoveredpermanent_failuresduplicates_suppressed
stripeinvoice.payment_succeeded4111
githubpush3101

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

Permanent failures by webhook event: static chart of synthetic permanent_failures values from the Measure webhook retry recovery example result
Download this SVG chart of the sample results for an article, runbook, or design review. Please credit Telemetry.

Reproduce the example

Download the sample data

The JSON bundle includes the event schema with field types, reproducible input rows, exact SQL, expected output, review notes, and engine version. The CSV contains the displayed result.

How the SQL works

  1. 1The first CTE collapses all attempts for one delivery into a single outcome record.
  2. 2A recovered delivery has at least one failed attempt and at least one successful attempt. A permanent failure never recorded success.
  3. 3Duplicate suppression is tracked separately because correct idempotency is a healthy outcome, not a processing failure.

Edge cases to check

  • Retain provider delivery IDs so retries can be grouped reliably.
  • A delivery may recover after the query window; use a window long enough to cover the provider's retry schedule.
  • Separate receipt from downstream side effects when success requires more than acknowledging the provider.

Recommended dashboard

  • Grouped bars: recovered and permanent_failures by event_type
  • Stat: duplicate suppression rate
  • Table: latest permanent failures with downstream job context

Alert guidance

Alert on permanent failures immediately for revenue-critical events, and on a sudden increase in recovered deliveries as an early-warning signal.

Read alert setup

Set up the events this query needs

Related instrumentation and guides

Define the source data

Event schemas for this analysis

Continue the analysis

Run it on your events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key