Telemetry
Data quality SQL recipe

Measure Required-Field Null Rate

Find event contracts where a required account, status, or correlation field is disappearing.

Beginnerproduct_eventsReviewed 2026-07-27Tested with Apache DataFusion 45.2.0

Question answered

Which event names have an unacceptable missing account identifier rate?

Schema compatibility does not guarantee analytical completeness. A nullable column can remain valid while a deployment stops populating the field needed for segmentation or joins.

Event contract

Fields the query expects

FieldTypeWhy it exists
timestamp_utcTimestampEvent occurrence time.
event_nameUtf8Stable event contract name.
account_idUtf8Account identifier required for these events.
producer_versionUtf8Application version that emitted the event.
DataFusion SQL

Copy the query

sql
SELECT
  event_name,
  producer_version,
  COUNT(*) AS events,
  SUM(CASE
    WHEN account_id IS NULL OR trim(account_id) = '' THEN 1
    ELSE 0
  END) AS missing_account_id,
  100.0 * SUM(CASE
    WHEN account_id IS NULL OR trim(account_id) = '' THEN 1
    ELSE 0
  END) / NULLIF(COUNT(*), 0) AS missing_rate_pct
FROM product_events
WHERE timestamp_utc >= now() - INTERVAL '24 hours'
GROUP BY event_name, producer_version
HAVING COUNT(*) >= 20
ORDER BY missing_rate_pct DESC, events DESC;

This read-only query is planned and executed against an empty typed table with Apache DataFusion 45.2.0. The deterministic sample output is synthetic and reviewed separately; validate field types, thresholds, and business definitions against your own data. Read the testing methodology.

Query result

Missing account ID rate by event and producer

The dashboard event has a version-specific completeness regression.

event_nameproducer_versioneventsmissing_account_idmissing_rate_pct
dashboard_viewedweb-18426,4201,18718.49
query_completedapi-9183,81070.18

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

How the SQL works

  1. 1The CASE expression treats both NULL and an empty string as missing.
  2. 2Grouping by producer_version makes deployment regressions easier to isolate.
  3. 3The minimum-volume rule prevents a handful of development events from dominating the report.

Edge cases to decide

  • Only evaluate a field on event contracts where it is truly required.
  • Anonymous product events may legitimately lack account_id and should use a separate completeness rule.
  • A placeholder such as unknown is not NULL but can still hide missing identity; measure sentinel values separately.

Recommended dashboard

  • Bars: missing_rate_pct by event_name and producer_version
  • Trend: completeness rate for critical contracts
  • Table: recent rows missing the required field

Alert guidance

Alert when a critical event contract exceeds its documented missing-field budget at meaningful volume.

Read alert setup

Put the recipe to work

Related instrumentation and guides

Continue the analysis

Run it on real events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key