Event Sampling Strategies for Structured Telemetry

Sampling keeps a subset of eligible events. It can reduce delivery and storage volume, but it also changes what the dataset can answer. Sampling should follow a measured cost or scale problem, not replace schema review, duplicate removal, or retention policy.

Before sampling, remove unused fields, normalize accidental high-cardinality values, stop duplicate producers, and measure event count and payload bytes with the telemetry volume recipe.

Preserve rare and important outcomes

Keep terminal failures, security-relevant events, billing records, incident milestones, and rare workflow outcomes at full fidelity unless an approved control says otherwise. A uniform one-percent sample can erase exactly the events an operator needs during an incident.

A common starting policy is:

• retain all failures, timeouts, retries exhausted, and explicit customer-impact events;
• retain all events from a small diagnostic allowlist during a bounded incident window;
• sample only high-volume successful outcomes;
• keep low-volume business milestones unsampled.

Sample deterministically

Deterministic sampling makes related decisions reproducible. Hash a stable identifier such as request_id, trace_id, or account_id with a versioned policy, then compare it with the target rate. The same identifier should make the same decision while the policy version remains unchanged.

Record:

• sampled or an equivalent collection outcome;
• sample_rate, such as 0.1;
• sampling_policy, such as api_success_v2;
• the stable dimension used for the decision.

Do not store the raw hash input when it is sensitive. Do not generate a new random decision independently for every step of a workflow if the analysis expects the steps to remain joinable.

Make weighted analysis explicit

If successful events are sampled at ten percent, each retained success represents approximately ten eligible successes under the policy assumptions. Store the inclusion probability and use an explicit weight when estimating counts.

SELECT
  route_template,
  SUM(1.0 / sample_rate) AS estimated_requests
FROM api_request_completed
WHERE timestamp_utc >= now() - INTERVAL '24 hours'
GROUP BY route_template
ORDER BY estimated_requests DESC;

Weighted counts do not automatically repair every statistic. Tail percentiles, distinct-user counts, funnels, cohort retention, and small customer segments can become unstable or biased. Keep unsampled data for analyses whose denominators or sequence relationships must be exact.

Version and evaluate the policy

Changing a rate changes the dataset. Record a policy version and effective time so queries can avoid comparing incompatible periods as though collection were constant.

Evaluate:

1. Retained event count and bytes.
2. Failure and rare-outcome coverage.
3. Estimated metric error against an unsampled fixture or temporary holdout.
4. Segment coverage for low-volume routes, plans, and regions.
5. The questions the sampled dataset can no longer answer.

Do not infer a product improvement from a volume drop that came from a collection-policy change.

Prefer retention changes when history is the problem

Sampling reduces future row coverage. Retention removes older data after a policy-defined period. If current incident detail must remain complete but long-term history is expensive, a shorter retention policy may be a better fit. Review data retention and deletion separately from sampling.

Continue with high-cardinality fields, telemetry cost management, and data-quality SQL recipes.