Production Instrumentation Checklist

Use this checklist after a synthetic event, query, dashboard, and alert work end to end. The goal is a bounded production rollout with an owner and fallback—not maximum event coverage.

Event contract

• The event name describes one durable workflow outcome.
• The business grain is documented: request, job, account, agent run, webhook, or another stable unit.
• Required and optional fields are listed with types and units.
• Status, error, feature, release, and environment values use bounded vocabularies.
• A stable identifier connects related events without exposing a secret.
• Retries and repeated attempts can be distinguished from final outcomes.

Review Designing an Event Schema, Canonical Wide Events, and Correlation IDs.

Sensitive-data boundary

• Passwords, API keys, authorization headers, cookies, and session tokens are prohibited.
• Raw request bodies, webhook payloads, prompts, completions, and job payloads stay out unless separately reviewed.
• Route templates replace URLs containing identifiers.
• Error categories replace raw private content where possible.
• Personal or regulated fields have an approved purpose, access boundary, and retention period.

Use Redacting Sensitive Data and the Security and Data Handling overview during review.

Credentials and environments

• Production ingestion uses a named, write-scoped key.
• Reporting automation uses a separate read-scoped key.
• Keys live in a secret manager or protected runtime configuration.
• Staging and production use separate keys and an explicit environment field.
• Rotation and revocation ownership is recorded.

Never introduce a required environment variable without confirming every deployed environment or providing a compatible fallback.

Delivery and failure behavior

• Telemetry delivery has a bounded timeout.
• A telemetry failure does not corrupt the primary application outcome.
• Retries are bounded and observable.
• Shutdown or flush behavior is tested for the selected SDK.
• High-volume loops do not emit an event for every internal step without an approved reason.

Run a controlled test with the Telemetry endpoint unavailable or slow. The application should follow its documented fallback.

Schema and data lifecycle

• Synthetic rows have been inspected in the final table.
• Field types remain compatible with existing producers and saved queries.
• Partition choices reflect real query filters.
• Retention matches the operational and privacy purpose.
• Row and table deletion responsibilities are understood.

See Schema Evolution, Choosing Partition Columns, and Data Retention and Deletion.

Analysis and response

• The first saved SQL result matches a known fixture.
• Denominators, exclusions, time windows, and identity rules are documented.
• Dashboard widgets lead back to inspectable SQL or rows.
• Every alert has a tested threshold, recipient, and response action.
• Commercial, security, or customer dimensions have an appropriate access boundary.

Use the SQL testing methodology and a related SQL recipe before treating the result as production evidence.

Roll out gradually

Start with one service, worker, or route. Observe event volume, schema behavior, query cost, delivery failures, and alert noise before expanding.

Record:

• Owner and review date.
• Deployed services and releases.
• Expected daily event volume.
• Saved query and dashboard links.
• Alert recipients and response documentation.
• Known gaps and the next review trigger.

If the workflow passes this checklist, expand from evidence. If it does not, keep the event in development or staging until the boundary is clear.