Telemetry
Webhooks SQL recipe

Measure Webhook Retry Recovery

Separate permanent webhook failures from deliveries that recovered on a later attempt.

Intermediatewebhook_deliveriesReviewed 2026-07-27Tested with Apache DataFusion 45.2.0

Reviewed by the Telemetry product team on . SQL compatibility, event contract, synthetic output, and operational caveats. Review standards and ownership

Question answered

Are webhook retries recovering failures or creating more work?

A raw failure count overstates customer impact when providers retry successfully. This query shows initial failures, recovered deliveries, permanent failures, and duplicate suppression together.

Event contract

Fields the query expects

FieldTypeWhy it exists
timestamp_utcTimestampWhen the processing attempt ended.
delivery_idUtf8Stable identifier shared by retry attempts.
providerUtf8Webhook provider.
event_typeUtf8Provider event type.
attemptInt64One-based processing attempt.
statusUtf8success, failed, or deduplicated.
DataFusion SQL

Copy the query

sql
WITH delivery_outcomes AS (
  SELECT
    delivery_id,
    provider,
    event_type,
    MAX(attempt) AS attempts,
    SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END) AS failed_attempts,
    SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END) AS successful_attempts,
    SUM(CASE WHEN status = 'deduplicated' THEN 1 ELSE 0 END) AS deduplicated_attempts
  FROM webhook_deliveries
  WHERE timestamp_utc >= now() - INTERVAL '7 days'
  GROUP BY delivery_id, provider, event_type
)
SELECT
  provider,
  event_type,
  COUNT(*) AS deliveries,
  SUM(CASE
    WHEN failed_attempts > 0 AND successful_attempts > 0 THEN 1 ELSE 0
  END) AS recovered,
  SUM(CASE
    WHEN failed_attempts > 0 AND successful_attempts = 0 THEN 1 ELSE 0
  END) AS permanent_failures,
  SUM(deduplicated_attempts) AS duplicates_suppressed
FROM delivery_outcomes
GROUP BY provider, event_type
ORDER BY permanent_failures DESC, recovered DESC;

This read-only query is planned and executed against an empty typed table with Apache DataFusion 45.2.0. The deterministic sample output is synthetic and reviewed separately; validate field types, thresholds, and business definitions against your own data. Read the testing methodology.

Query result

Permanent failures by webhook event

Recovered events remain visible without being counted as permanent customer impact.

providerevent_typedeliveriesrecoveredpermanent_failuresduplicates_suppressed
stripeinvoice.payment_succeeded4111
githubpush3101

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

Permanent failures by webhook event: static chart of synthetic permanent_failures values from the Measure Webhook Retry Recovery example result
Indexable SVG of the deterministic example output. Download it for an article, runbook, or design review with attribution.

Reproduce the example

Download the public fixture

The JSON bundle includes the typed event contract, reproducible input rows, exact SQL, expected output, review notes, and engine version. The CSV contains the displayed result.

How the SQL works

  1. 1The first CTE collapses all attempts for one delivery into a single outcome record.
  2. 2A recovered delivery has at least one failed attempt and at least one successful attempt. A permanent failure never recorded success.
  3. 3Duplicate suppression is tracked separately because correct idempotency is a healthy outcome, not a processing failure.

Edge cases to decide

  • Retain provider delivery IDs so retries can be grouped reliably.
  • A delivery may recover after the query window; use a window long enough to cover the provider's retry schedule.
  • Separate receipt from downstream side effects when success requires more than acknowledging the provider.

Recommended dashboard

  • Grouped bars: recovered and permanent_failures by event_type
  • Stat: duplicate suppression rate
  • Table: latest permanent failures with downstream job context

Alert guidance

Alert on permanent failures immediately for revenue-critical events, and on a sudden increase in recovered deliveries as an early-warning signal.

Read alert setup

Put the recipe to work

Related instrumentation and guides

Define the source data

Event schemas for this analysis

Continue the analysis

Run it on real events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key