Database Reliability Monitoring with SQL

Database symptoms often appear in the application before they are obvious in a server metric: callers wait for a connection, one operation fingerprint slows down, transactions roll back, locks block user work, a replica falls behind, or a migration fails during a release. Structured application events connect those outcomes with the service, release, route, tenant impact, and controlled error category needed to respond.

Emit bounded events at the application boundary

Record one completion event for each database operation or logical transaction. Use a controlled operation name or normalized fingerprint instead of raw SQL. Include duration, outcome, database role, service, release, and a privacy-reviewed account identifier only when it supports impact analysis.

Never log query parameters, connection strings, credentials, authorization data, or unrestricted SQL text. Error categories such as deadlock, serialization_failure, and connection_timeout are safer and easier to group than raw exception messages.

{
  "event_name": "database_query_completed",
  "query_fingerprint": "checkout.select_with_line_items",
  "service": "checkout-api",
  "database_name": "app_production",
  "status": "success",
  "duration_ms": 842,
  "rows_returned": 4,
  "release": "2026.07.2",
  "environment": "production"
}

Use the node-postgres integration or Prisma integration as a starting point, then centralize the wrapper so every operation uses the same field names, clock, status values, and redaction policy.

Separate the reliability questions

One generic “database health” score hides different failure modes. Use focused event tables or stable event names for:

Query completion: duration, normalized fingerprint, rows, outcome, and release.
Pool state: active, idle, configured maximum, acquisition wait, and timeout.
Transactions: logical transaction identifier, commit or rollback, and controlled error type.
Lock waits: blocked and blocking fingerprints, wait duration, resolution, and deadlock detection.
Replication or CDC: consumer, region, seconds and bytes behind, and current status.
Migrations: migration identifier, release, duration, terminal status, and rollback category.

The database reliability recipe collection contains a schema, tested query, deterministic result, visualization, edge cases, dashboard plan, and alert guidance for each question. You can run the included fixtures in the read-only browser SQL playground before adapting them.

Map OpenTelemetry database fields deliberately

If the application already emits OpenTelemetry database spans, reuse the stable meaning of fields instead of creating a competing vocabulary. The current OpenTelemetry database client span conventions define db.system.name, low-cardinality db.operation.name, db.namespace, db.collection.name, db.query.summary, and database response status context. They also warn that query text can be high-cardinality and requires sanitization.

A practical structured-event mapping is:

OpenTelemetry context	Structured event field	Review note
`db.system.name`	`database_system`	Keep a bounded database product identifier.
`db.operation.name`	`operation_name`	Use a controlled verb such as `SELECT` or a stable client operation.
`db.query.summary`	`query_fingerprint`	Prefer a low-cardinality summary generated before collection.
`db.response.status_code`	`database_status_code`	Preserve the driver or database code only when its semantics are documented.
span duration	`duration_ms`	State whether it includes pool acquisition and network time.
trace and span context	`trace_id`, `span_id`	Use identifiers for correlation without copying span payloads.

The mapping is not automatic proof that a field is safe. Query summaries, collection names, namespaces, and response messages can still expose tenant or schema details in some systems. Review the actual emitted values, cap cardinality, and keep raw query text and bound values out by default.

Investigate in order

Begin with customer-visible duration and failure rate, then check whether pool acquisition explains the latency. Rank operations by both p95 duration and total query time: a moderately slow query executed thousands of times can consume more application time than a rare outlier. Compare SQLSTATE or another controlled driver code by operation and release instead of grouping raw messages.

If failures are transactional, split expected retryable rollbacks from terminal errors and review long-running transaction classes separately. Inspect lock-wait and deadlock events when concurrent writers are involved. Compare connection opens, closes, acquisition timeouts, and affected requests before interpreting pool saturation as a sizing problem. Check replica or CDC lag beside application-observed stale reads and failover outcomes before trusting a downstream read model. Align migration failures with deployment timestamps.

Do not increase a connection-pool limit from utilization alone. A larger pool can move contention into the database. Require waiting callers or timeouts, confirm database capacity, and monitor the change.

Know the application-side boundary

These events answer which application workflow, release, region, or customer-facing request experienced a database symptom. They do not replace the database's own diagnostic systems. Use database-native tools for:

Query plans, optimizer estimates, buffer and cache behavior, table statistics, and vacuum or compaction state.
Server wait events, lock graphs, active-session inspection, storage latency, and resource saturation.
Replication topology, write-ahead-log retention, failover orchestration, backup verification, and point-in-time recovery.
Authoritative audit, access-control, encryption, and compliance evidence required by the database or security program.

Join the two views during investigation: use structured application events to locate impact and ownership, then use restricted database diagnostics to explain the server-side cause. Avoid copying sensitive server diagnostics into a broad analytics table merely to make the join convenient.

Turn validated queries into operations

Dashboards should keep volume beside every rate, use complete UTC buckets, and preserve a path from the aggregate to recent safe events. Alerts need sustained conditions, minimum volume, an owner, and a documented response. A single slow query, brief lag spike, or expected rollback rarely justifies a page.

Test synthetic success, timeout, deadlock, duplicate, and delayed-event cases before relying on a query. Record the event version and threshold beside the dashboard. The SQL methodology describes the automated recipe checks; the database reliability use case connects the resulting signals to an implementation workflow.

Use this doc with your coding agent