Event Delivery, Idempotency, and Duplicate Handling

Sending an event over a network creates an outcome the producer may not be able to observe perfectly. A timeout can happen before the server receives a request, while it processes the request, or after it accepts the event but before the response reaches the caller. Retrying improves delivery, but it can also create duplicate rows.

Treat delivery policy as part of the event contract rather than an incidental SDK setting.

Give every logical event a stable identity

Create one event_id when the application outcome becomes known. Reuse that identifier for every network attempt that represents the same outcome. Do not generate a new identifier inside the retry loop.

const eventId = crypto.randomUUID();
const event = {
  event_id: eventId,
  job_id: job.id,
  status: "failed",
  error_type: "provider_timeout",
  attempt: job.attempt,
};

await sendWithBoundedRetry("job_completed", event);

An event_id makes duplicates detectable; it does not by itself guarantee that storage will reject every duplicate. If the business workflow requires exactly-once effects, enforce that requirement in the system that owns the effect. Telemetry should describe the result, not become the transaction coordinator for a payment, entitlement, or database mutation.

Decide which failures are retryable

Retry temporary capacity and transport failures such as connection errors, 429, 502, 503, and 504. Respect Retry-After when the response provides it. Use exponential backoff with jitter, and cap both the number of attempts and total elapsed time.

Do not retry an unchanged 400 request. Invalid JSON, table names, field types, or schemas require a producer change. Replace a missing or revoked credential after 401; use a key with the required scope after 403.

The rate-limits and API-errors reference contains the response classes and retry rules.

Keep telemetry off the critical path by default

For normal product and operational analytics, a temporary telemetry failure should not turn a successful customer request into an error. Bound the telemetry timeout, report a safe diagnostic category, and continue according to the product's failure policy.

Some workflows need stronger delivery:

Usage records that feed billing reconciliation.
Security audit events required by an approved control.
Irreversible business outcomes with no other source of truth.

For those cases, write the outcome to a durable application-owned outbox in the same transaction as the business change. A worker can deliver the event later while preserving its original event_id, occurrence time, and schema version.

Measure duplicate and missing delivery

Use the duplicate event IDs SQL recipe to find identifiers with more than one stored row. Break duplicates down by producer, release, and retry reason so the fix targets the actual delivery path.

Also monitor:

accepted and rejected event counts by producer;
age of the oldest undelivered outbox record;
attempts per logical event;
permanent failures after the retry budget;
the delay between occurred_at and timestamp_utc.

Deduplicating every query can hide a broken producer. Keep duplicate-rate monitoring visible even if a downstream report selects one row per event_id.

Test the ambiguous cases

Exercise a success, a definite rejection, a connection failure before delivery, and a timeout after the server may have accepted the event. Confirm that retry attempts reuse the event ID and that application responses follow the documented failure policy.

Continue with event-ingestion troubleshooting, schema evolution, and telemetry cost management.

Use this doc with your coding agent