How Telemetry evaluates SQL alerts

A useful alert is more than a threshold attached to a chart. The evaluator must decide when a query is due, which SQL to run, which rows count, how to reduce those rows to one value, whether the state changed, and when to notify.

This guide documents the evaluation path implemented by Telemetry. It is an inspectable behavior description, not an uptime, latency, or delivery guarantee.

Evaluation sequence

The evaluator uses a chained one-minute scheduling loop. Each pass loads enabled alerts, evaluates them concurrently with isolated failure handling, records the pass result, and schedules the next pass after the current pass finishes. The one-minute loop is a polling cadence; each alert still has its own configured interval.

For every enabled alert:

Check whether the alert is due from its last evaluation time and configured interval.
Claim the alert using its current version so another evaluator cannot claim the same version concurrently.
Resolve the query: use the exact saved SQL for a query-backed alert, or regenerate SQL from the stored Explore configuration.
Reject query-backed SQL containing write statements.
Execute the query and inspect the result columns and rows.
Optionally drop the newest row when it represents an incomplete time bucket.
Take the configured number of recent points and aggregate them into one observed value.
Compare that value with the threshold and operator.
Store the new state, observed value, evaluation time, and any controlled error.
Add history and attempt email delivery only when the state transitions.

This sequence matters when troubleshooting. A valid query can still produce no usable value, a usable value can remain below the threshold, and a breached threshold can remain silent when the alert was already firing.

Saved query and Explore sources

A query-backed alert runs the SQL saved with the alert. The evaluator does not silently replace it with a newer query revision. Review the SQL attached to the alert when a dashboard and alert appear to disagree.

An Explore-backed alert stores the Explore configuration and regenerates SQL through the same query-building boundary used by Explore. Changes to schema, filters, grouping, or aggregation can therefore affect whether that stored configuration still resolves.

Write-oriented statements are rejected for query-backed alerts. Alerts are intended to observe results, not mutate tables or administrative state. Keep alert SQL bounded by time and result size even when the SQL is read-only.

Newest-point exclusion

Time-bucketed queries commonly return the current, incomplete bucket first. Comparing that partial minute or hour with a full historical bucket can create false recovery or false breach signals.

When exclude latest incomplete point is enabled, the evaluator sorts rows newest first and removes the newest result row before selecting the configured number of points. That setting is correct only when each row truly represents a time bucket and the newest row is incomplete.

Do not enable it for a single-row aggregate such as:

SELECT
  100.0 * SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END) / COUNT(*) AS failure_rate
FROM background_job_completed
WHERE completed_at >= now() - INTERVAL '15 minutes';

Dropping the only row leaves no value to evaluate. For a time series, return an explicit bucket column and one numeric value column:

SELECT
  date_bin(INTERVAL '5 minutes', completed_at, TIMESTAMP '1970-01-01') AS bucket,
  100.0 * SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END) / COUNT(*) AS failure_rate
FROM background_job_completed
WHERE completed_at >= now() - INTERVAL '35 minutes'
GROUP BY bucket
ORDER BY bucket DESC;

Point selection and aggregation

After newest-point handling, the evaluator takes the configured number of recent rows. It extracts the selected numeric value and reduces those points using the alert aggregation setting.

Use a single point when the SQL already calculates the complete decision window. Use multiple points when the alert intentionally requires persistence across several buckets. The aggregation should match the operational question:

maximum asks whether any selected point breached.
minimum asks whether every selected point stayed above a floor.
average smooths short variation but can hide one severe point.
sum fits counts accumulated across non-overlapping buckets.
latest uses the newest remaining complete point.

Document this choice in the response procedure. “Failure rate above 5” is ambiguous unless the team knows whether that means the latest five-minute bucket, the average of three buckets, or the maximum of those buckets.

State transitions and notifications

Telemetry stores alert state separately from a single evaluation. A successful evaluation can produce ok or firing; an execution, extraction, or configuration problem produces an error result.

History and notification are transition-oriented. A move from ok to firing creates a trigger transition. A move from firing to ok creates a resolution transition. Repeated evaluations in the same state update the evaluation record without sending the same transition email on every polling pass.

Current notification delivery for this evaluator is email. Delivery is attempted after the state transition is stored. A delivery failure must not erase the fact that the alert transitioned, which is why alert history and email delivery should be investigated separately.

Concurrency and claim behavior

Each alert is claimed with optimistic concurrency using its current version. If the version has changed or another evaluator already claimed that version, the claim does not succeed. This prevents two evaluators from knowingly processing the same alert version at the same time.

The scheduler evaluates the enabled alert set with settled promises, so one rejected alert does not prevent the remaining alerts in that pass from completing. The loop schedules the next pass only after the current pass has settled, avoiding overlapping passes inside one process.

These controls explain evaluator behavior; they are not a distributed-service availability claim. Deployment topology, process restarts, email-provider behavior, and future implementation changes still belong in operational review.

Troubleshooting checklist

When an alert does not behave as expected:

Run the exact attached SQL and confirm it is read-only.
Check that the selected value column is numeric and exists in every relevant row.
Inspect result ordering; newest-point logic depends on knowing which row is newest.
Confirm whether the query returns one aggregate row or multiple time buckets.
Review exclude latest incomplete point, point count, and aggregation together.
Verify the operator and threshold use the same unit as the selected value.
Check alert history for the most recent state transition and controlled error.
Separate query/evaluation failures from email-delivery failures.
Confirm the alert is enabled and enough time has elapsed for its configured interval.
Test with a synthetic event fixture on both sides of the threshold.

For delivery-specific investigation, continue with Alert Delivery Troubleshooting. For event and query design, use Alerts and Create Your First Dashboard and Alert.

Interpretation boundary

The evaluator can execute the configured decision consistently, but it cannot decide whether the business definition is correct. The event grain, denominator, time window, threshold, missing-data policy, and response owner remain part of the alert contract.

Re-review an alert after schema changes, query edits, release changes, traffic shifts, and incident retrospectives. A technically functioning alert with an obsolete threshold or denominator is still an unreliable alert.

Use this doc with your coding agent