Test Telemetry Instrumentation in CI

Telemetry instrumentation is application behavior and should be tested like any other integration boundary. A useful CI suite proves that the right terminal outcome creates the right event, prohibited content stays out, delivery failure does not corrupt the application outcome, and operational SQL still returns the expected result.

Do not make CI depend on the production Telemetry API. Inject an event sink or HTTP transport and capture payloads in memory.

Test the event contract

Build events in one function so schema rules are reviewable:

type CheckoutInput = {
  checkoutId: string;
  status: "success" | "failed";
  durationMs: number;
  errorType?: "payment_declined" | "provider_timeout";
};

export function checkoutEvent(input: CheckoutInput) {
  return {
    event_name: "checkout_completed",
    schema_version: 1,
    checkout_id: input.checkoutId,
    status: input.status,
    duration_ms: input.durationMs,
    ...(input.errorType ? { error_type: input.errorType } : {}),
  };
}

Unit tests should assert exact field names and types, controlled category values, explicit units, and conditional requirements. Prefer exact-object assertions over snapshots that reviewers may update without noticing a semantic change.

Exercise every terminal path

For the owning workflow, cover:

• success
• expected business failure
• dependency timeout
• retry followed by success
• retries exhausted
• cancellation or human handoff where applicable
• duplicate delivery
• missing optional context

Assert the grain. A logical request event should be emitted once even if the implementation retries internally. An attempt event should include an attempt number and a stable logical-operation ID.

Enforce the privacy boundary

Create hostile fixtures containing an authorization header, cookie, email, raw URL query, request body, prompt, tool argument, and exception message. Assert that none can reach the captured event.

An allowlist is easier to test than a growing denylist. If redaction is a fallback layer, test it separately and ensure an unknown nested field cannot bypass it.

Test delivery failure

Make the fake sink reject, time out, and return a rate-limit response. Confirm the application follows its documented policy:

• best-effort analytics does not turn a successful customer operation into an error
• critical audit or billing events use the approved durable path
• retries are bounded and use idempotency where required
• shutdown flushes respect a deadline
• telemetry failures are observable without recursively logging the failed payload

See Event Delivery and Idempotency and Batching, Backpressure, and Shutdown.

Validate SQL with deterministic fixtures

Store a small synthetic dataset and the expected rows for important SQL. Include success, failure, duplicate, null, late, and boundary-timestamp cases. A query test should make its counting rules visible.

The SQL recipe library includes deterministic input rows and expected output, while the SQL playground can run supported fixtures locally. Use those patterns for application-owned dashboards and alerts.

Add deployment verification

CI proves code behavior, not runtime configuration. After deployment to a non-production environment:

1. Send a uniquely identified synthetic event.
2. Verify the HTTP response and stored table.
3. Confirm field types and schema version.
4. Run the smallest query that finds the event.
5. Delete or expire the fixture according to policy.

Do not fail application startup merely because a newly introduced analytics variable is absent. Roll configuration out first, retain the existing behavior as a fallback, and enforce only after every deployed environment is verified.

Document the contract in an event tracking plan, follow the production instrumentation checklist, and use Troubleshooting Event Ingestion when deployment verification fails.