Find Duplicate Event IDs

Event contract

Fields the query expects

Field	Type	Why it exists
timestamp_utc	Timestamp	Event occurrence time.
event_id	Utf8	Stable identifier reused for delivery retries.
event_name	Utf8	Event contract name.
source	Utf8	Producer name.

DataFusion SQL

Copy the query

sql

SELECT
  event_id,
  event_name,
  source,
  COUNT(*) AS deliveries,
  MIN(timestamp_utc) AS first_seen_at,
  MAX(timestamp_utc) AS last_seen_at
FROM telemetry_events
WHERE timestamp_utc >= now() - INTERVAL '7 days'
  AND event_id IS NOT NULL
GROUP BY event_id, event_name, source
HAVING COUNT(*) > 1
ORDER BY deliveries DESC, last_seen_at DESC
LIMIT 100;

This read-only query is planned and executed against an empty typed table with Apache DataFusion 45.2.0. The deterministic sample output is synthetic and reviewed separately; validate field types, thresholds, and business definitions against your own data. Read the testing methodology.

Query result

Repeated deliveries by event ID

The invoice event was delivered four times and needs idempotent counting downstream.

evt_8f31

evt_77ac

event_id	event_name	source	deliveries	first_seen_at	last_seen_at
evt_8f31	invoice_paid	stripe_webhook	4	12:01:03	12:06:44
evt_77ac	job_completed	billing_worker	2	10:14:22	10:14:39

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

How the SQL works

1Grouping by event_id exposes retry deliveries without assuming that events close in time are duplicates.
2Including event_name and source catches producers that accidentally reuse an identifier across different contracts.
3The first and last timestamps show whether duplicates arrive in a short retry burst or much later.

Edge cases to decide

Do not generate a new event ID for every retry if downstream consumers need idempotency.
A reused identifier across unrelated events is an instrumentation defect, not a harmless duplicate.
If duplicates are removed before storage, emit a separate deduplication outcome so the retry behavior remains observable.

Recommended dashboard

Stat: duplicate event IDs in the selected period
Bars: duplicate deliveries by source
Table: newest repeated identifiers

Alert guidance

Alert when duplicate volume or the maximum delivery count rises above the normal retry baseline.

Read alert setup

Put the recipe to work

Related instrumentation and guides

Data quality SQL recipes Webhook duplicate recipe Correlation ID guidance

Continue the analysis

Measure Webhook Latency and Duplicate Rate

Compare processing latency, duplicate deliveries, and failures by webhook provider and event type.

Open recipe

Measure Event Ingestion Freshness

Find event sources that stopped delivering data or are arriving substantially later than they occurred.

Open recipe

Measure Required-Field Null Rate

Find event contracts where a required account, status, or correlation field is disappearing.

Open recipe

Run it on real events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key