Telemetry
API reliability SQL recipe

Compare Dependency p95 Latency

Find databases, APIs, caches, and queues contributing the most tail latency to requests.

Intermediatedependency_callsReviewed 2026-07-27Tested with Apache DataFusion 45.2.0

Question answered

Which downstream dependencies have the worst tail latency and failure rate?

A slow route is easier to fix when dependency timing is attached to the completed unit of work. Tail latency reveals intermittent downstream pressure that averages hide.

Event contract

Fields the query expects

FieldTypeWhy it exists
timestamp_utcTimestampDependency call completion time.
dependency_nameUtf8Stable logical dependency name.
operationUtf8Normalized operation or route.
statusUtf8success, failed, or timeout.
latency_msFloat64Dependency duration in milliseconds.
DataFusion SQL

Copy the query

sql
SELECT
  dependency_name,
  operation,
  COUNT(*) AS calls,
  approx_percentile_cont(latency_ms, 0.50) AS p50_ms,
  approx_percentile_cont(latency_ms, 0.95) AS p95_ms,
  100.0 * SUM(CASE WHEN status <> 'success' THEN 1 ELSE 0 END)
    / NULLIF(COUNT(*), 0) AS failure_rate_pct
FROM dependency_calls
WHERE timestamp_utc >= now() - INTERVAL '24 hours'
GROUP BY dependency_name, operation
HAVING COUNT(*) >= 50
ORDER BY p95_ms DESC
LIMIT 20;

This read-only query is planned and executed against an empty typed table with Apache DataFusion 45.2.0. The deterministic sample output is synthetic and reviewed separately; validate field types, thresholds, and business definitions against your own data. Read the testing methodology.

Query result

Dependency p95 latency

Invoice history has a severe long tail despite an acceptable median.

dependency_nameoperationcallsp50_msp95_msfailure_rate_pct
billing_databaseinvoice_history1,840841,8200.49
search_providerdocument_search6,2201327601.21
redissession_lookup44,1003120.02

Synthetic example output. Run the query against your own event schema and thresholds before using it for operational decisions.

How the SQL works

  1. 1Grouping by logical dependency and operation keeps the result actionable.
  2. 2The gap between p50 and p95 distinguishes consistently slow services from intermittent tail behavior.
  3. 3Failure rate prevents a fast-failing dependency from appearing healthy.

Edge cases to decide

  • Avoid raw SQL text, full URLs, or customer identifiers in operation names.
  • Parallel dependency calls do not add linearly to request duration.
  • If traces already contain this timing, record only the business context needed to connect the outcome.

Recommended dashboard

  • Grouped bars: p50_ms and p95_ms by dependency
  • Trend: p95 latency and failure rate
  • Table: slow dependency calls with route, release, and trace ID

Alert guidance

Alert when a critical dependency exceeds its p95 objective and is affecting a minimum number of calls.

Read alert setup

Put the recipe to work

Related instrumentation and guides

Continue the analysis

Run it on real events

Create a table, adapt the fields, and save the result

Start free, send structured events, and use the query result as a chart, shared dashboard widget, or alert input.

Get an API key