Structured log management and SQL

Definition

What structured log management changes

Debug logs record individual steps. To find out whether a checkout finished or a retry recovered, you may need to piece together several messages. A structured event puts the final result and the fields needed to investigate it in one record.

Structured log management means defining event fields, deciding which data to collect, and storing events in queryable tables. Use those tables for searches, SQL, charts, dashboards, and alerts. Keep traces, metrics, and debug logs for questions that need them.

Include fields you need for a query

Choose fields based on the questions you need to answer. Avoid copying entire payloads. They add storage cost and may expose private data.

Text logs and structured events

What each record contains

Concern	Message-oriented logs	Structured events
Primary unit	One message about a local code path	One completed workflow or meaningful state change
Context	Often spread across many lines and services	Stable identifiers, outcome, duration, and dimensions together
Analysis	Search and parsing patterns	Typed filters, groups, joins, percentiles, funnels, and cohorts
Schema	Implicit in prose and formatting	Named fields with explicit types, units, and allowed values

JSON

A completion event

json

{
  "event": "checkout_completed",
  "timestamp": "2026-07-28T18:42:16Z",
  "request_id": "req_01K1A9",
  "account_id": "acct_812",
  "plan": "growth",
  "status": "success",
  "duration_ms": 842,
  "amount_usd": 129.00,
  "payment": {
    "provider": "stripe",
    "attempt": 1
  }
}

Event schema

Keep events consistent as your app changes

Choose when the event should fire

Emit an event when a request, job, webhook, agent run, billing change, or product milestone reaches an outcome someone may need to explain.

Capture enough context once

Include a consistent event name, UTC time, status, duration, environment, release, and approved identifiers. Add other fields only when they answer a known question.

Preserve types and units

Keep numbers as numbers, booleans as booleans, and units in field names. Avoid parsing latency, money, or counts from message strings later.

Exclude sensitive content

Exclude secrets, credentials, authorization headers, raw prompts, payment details, and unnecessary personal data. Prefer categorized error context.

DataFusion SQL

Query checkout results over time

sql

SELECT
  date_trunc('hour', timestamp_utc) AS hour,
  COUNT(*) AS checkouts,
  SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END) AS succeeded,
  ROUND(
    100.0 * SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END)
      / NULLIF(COUNT(*), 0),
    2
  ) AS success_rate_pct,
  approx_percentile_cont(duration_ms, 0.95) AS p95_duration_ms
FROM checkout_events
WHERE timestamp_utc >= now() - INTERVAL '24 hours'
GROUP BY date_trunc('hour', timestamp_utc)
ORDER BY hour;

Read the trend and the denominator together

14:00

99.4%

15:00

98.9%

16:00

94.7%

17:00

97.8%

The chart shows a change at 16:00. The query also returns checkout count and p95 duration, so you can check the sample size and latency. Once you have enough checkouts to compare, group that hour by release, provider, plan, or account.

Investigation sequence

1.Detect a meaningful change in rate, latency, cost, or volume.
2.Confirm the time window, denominator, and event freshness.
3.Group the results by service owner, release, or rollout group to find where the change happened.
4.Inspect correlated events for the affected requests or accounts.
5.Save the validated query and document the response.

Start with one workflow

Send a synthetic event and answer the first question

Send test data and check the stored fields before adding events across your application.

Explore structured events

Query application results with structured events