Skip to content
Data quality SQL recipe

Catch event-field renames that hide rows from averages

Compare an old timing-field average with schema-aware coverage so an event rename cannot silently drop completed jobs.

Intermediatejob_finishedReviewed 2026-09-30Tested with Apache DataFusion 45.2.0

Reviewed by the Telemetry product team on . We checked the SQL syntax, required event fields, sample results, and limits on using the query. Who reviews this page

Question answered

Did a field rename make the duration average ignore completed jobs?

A query can keep returning a plausible average after instrumentation changes. Count all completed events, map each documented schema version to its timing field, and show the share with a usable value beside the average.

Event schema

Fields the query expects

FieldTypeWhy it exists
timestamp_utcTimestampUTC finish-event time; use a fixed analysis window in production.
job_idUtf8One fictional completed logical job per row.
schema_versionInt64Version of the event contract; 1 uses duration_ms and 2 uses latency_ms.
duration_msInt64Release-1 duration, null on release 2.
latency_msInt64Release-2 duration, null on release 1.
DataFusion SQL

Copy the query

sql
WITH normalized AS (
  SELECT
    job_id,
    schema_version,
    duration_ms,
    CASE schema_version
      WHEN 1 THEN duration_ms
      WHEN 2 THEN latency_ms
    END AS measured_ms
  FROM job_finished
)
SELECT
  COUNT(*) AS completed_jobs,
  COUNT(duration_ms) AS jobs_in_old_average,
  ROUND(AVG(duration_ms), 1) AS old_average_ms,
  COUNT(measured_ms) AS jobs_with_duration,
  SUM(CASE WHEN measured_ms IS NULL THEN 1 ELSE 0 END) AS unmeasured_jobs,
  ROUND(100.0 * COUNT(measured_ms) / NULLIF(COUNT(*), 0), 1)
    AS duration_coverage_pct,
  ROUND(AVG(measured_ms), 1) AS schema_aware_average_ms
FROM normalized;

This read-only query is planned and executed against an empty typed table with Apache DataFusion 45.2.0. We review the synthetic sample output separately. Check field types, thresholds, and counting rules against your own data. Read the testing methodology.

Query result

A plausible average can cover only half the jobs

The old field averages two fictional jobs at 475 ms. The version-aware query covers all four at 580 ms.

completed_jobsjobs_in_old_averageold_average_msjobs_with_durationunmeasured_jobsduration_coverage_pctschema_aware_average_ms
4247540100580

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

A plausible average can cover only half the jobs: static chart of synthetic jobs_with_duration values from the Catch event-field renames that hide rows from averages example result
Download this SVG chart of the sample results for an article, runbook, or design review. Please credit Telemetry.

Reproduce the example

Download the sample data

The JSON bundle includes the event schema with field types, reproducible input rows, exact SQL, expected output, review notes, and engine version. The CSV contains the displayed result.

How the SQL works

  1. 1The old query AVG(duration_ms) still returns 475 ms because SQL averages ignore nulls. Only two of four completed jobs contribute to it.
  2. 2The version-aware CASE maps release 1 to duration_ms and release 2 to latency_ms, giving 580 ms across all four fictional jobs.
  3. 3Keep duration coverage beside the average. An unknown version or a missing field can leave the average unchanged while coverage falls.

Edge cases to check

  • Use a fixed UTC time window when comparing releases; this four-row lesson leaves all fictional rows in scope.
  • This example assumes one finish row per completed logical job. Deduplicate repeated finish events by job_id before using it for a real job count.
  • A field rename is comparable only if both versions time the same start and stop points in the same unit.
  • A missing or unknown schema_version maps to null and lowers coverage; do not silently coalesce it to an old field.
  • The four rows are fictional and do not describe a customer, a real incident, or measured product performance.

Recommended dashboard

  • Table: completed jobs, duration coverage, and average by release
  • Trend: duration coverage percentage over deploy time

Alert guidance

Alert on falling duration coverage only after establishing an expected minimum event count and documenting the version mapping.

Read alert setup

Set up the events this query needs

Related instrumentation and guides

Continue the analysis

Run it on your events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key