Troubleshooting Event Ingestion

When an event does not appear where expected, separate request delivery, authentication, payload validation, schema compatibility, and query freshness. A successful application operation does not prove that telemetry ingestion succeeded, and an accepted HTTP request does not prove that the later query is looking at the same table and time range.

Use a synthetic event with a unique safe identifier while diagnosing. Do not copy a production payload into logs, tickets, or command history.

1. Capture the actual HTTP result

Temporarily send one event with cURL so you can see the response status and body:

curl -i -X POST https://api.telemetry.sh/log \
  -H "Content-Type: application/json" \
  -H "Authorization: $API_KEY" \
  -d '{
    "table": "ingestion_diagnostics",
    "data": {
      "event_id": "diagnostic-2026-07-28-01",
      "source": "manual_check",
      "status": "expected"
    }
  }'

Keep the API key in an environment variable. Do not paste it into the payload or print it while collecting diagnostics.

Interpret the status before retrying:

400 means the JSON, table name, timestamp, data shape, or field type must change. Retrying the same body will not repair it.
401 means the key is missing, malformed, invalid, or revoked.
403 means the key does not have write scope for the operation.
429 means the current request rate or quota was exceeded. Respect Retry-After when supplied and use bounded exponential backoff with jitter.
5xx represents a server-side failure for an otherwise valid request. Retry a limited number of times without blocking the main application indefinitely.

Read Rate Limits and API Errors for the complete client policy.

2. Confirm the destination table

The Log API normalizes the requested table name: spaces become underscores and letters become lowercase. After normalization, only lowercase ASCII letters, numbers, and underscores are valid.

For example, Checkout Events becomes checkout_events. Querying a guessed name such as CheckoutEvents will not inspect the same destination.

During diagnosis, use a simple explicit name in the request and then inspect the table list or schema in Telemetry. If two services should write to one table, make sure both use the same normalized name and field types.

3. Validate the accepted data shape

The data property may be:

one JSON object
an array of JSON objects
a JSON string that decodes to an object
an array containing objects and JSON strings that each decode to objects

Top-level numbers, booleans, null, and strings that decode to those values are rejected. Arrays cannot contain arbitrary scalar values. Deeply nested payloads beyond the documented limit are also rejected.

Reduce a failing event to three harmless fields. Add fields back in small groups until the request fails again. This isolates the invalid shape without exposing the original customer payload.

4. Check timestamp behavior

Telemetry adds a UTC timestamp when it is missing or null. Unix timestamp integers and numeric strings are interpreted as Unix seconds and normalized to RFC 3339 when they are within the supported range. A client-provided timestamp_utc is removed because that field is managed by the query layer.

If a fresh event appears outside the query window:

remove the custom timestamp and send a new synthetic event
query by the generated timestamp_utc
compare the application clock, source timestamp, and query time zone
check whether the original timestamp was accidentally milliseconds rather than seconds

Use Working with Timestamps when the source event time must be preserved separately from receipt time.

5. Inspect schema compatibility

The first accepted events establish field types. Adding a new optional field is different from changing an existing field from a number to a string, boolean, timestamp, or nested object.

Inspect the table schema and compare the failing payload field by field. Common drift includes:

an identifier sent as an integer by one producer and a string by another
a duration sent as a number in one release and "842ms" in another
a nested object replaced with a scalar
a money value changing between integer cents and decimal currency units
a status changing type because an SDK serializes an enum differently

When a concept genuinely changes type or unit, add a versioned field name and migrate queries deliberately. Read Event Data Types and Nullability and Schema Evolution.

6. Isolate bulk failures

For a rejected batch, reproduce with a small synthetic subset. If needed, split the batch until the incompatible item is identified. Preserve a stable event_id while retrying the same logical event so duplicate delivery can be measured.

Do not silently assign a new identifier to every network attempt. That turns one outcome into several rows and makes retry recovery, billing totals, and funnels unreliable. Audit duplicates with the duplicate event ID SQL recipe.

7. Verify the event with SQL

Query the exact diagnostic identifier and a generous UTC range:

SELECT
  event_id,
  source,
  status,
  timestamp_utc
FROM ingestion_diagnostics
WHERE event_id = 'diagnostic-2026-07-28-01'
  AND timestamp_utc >= now() - INTERVAL '24 hours'
ORDER BY timestamp_utc DESC;

If the row is present but a dashboard is empty, compare the dashboard's table, filters, time range, and expected field types. If no fresh rows from an entire source appear, use the ingestion freshness recipe to make the gap visible.

Prevention checklist

keep server-side keys out of browser bundles and scope them to the required operations
capture safe status, endpoint, request ID, and error category for ingestion failures
use stable table names, event names, field types, and explicit units
send schema changes through a synthetic or staging check before production
cap retries so telemetry cannot exhaust workers or alter a completed customer response
monitor freshness and duplicate identifiers separately from the business dashboard

See the Log API for exact normalization rules and the structured logging guide for safer event-contract design.

Use this doc with your coding agent