Choosing Partition Columns

Partition columns can make selective queries much faster by letting Telemetry skip parts that cannot contain the requested values. They are especially useful for agents and automated workflows that repeatedly investigate one customer, service, environment, trace, or event type in a large table.

They are not the same as SQL PARTITION BY, and they do not change query results. Telemetry hashes the configured values when it writes data and records that partition information with each part. At query time, a compatible filter is hashed the same way, so unrelated parts can be excluded before their data is read.

When to use them

Partition a table when all of these are true:

the table has enough data or parts that scanning it is material work
important queries repeatedly filter on the same field
those filters usually select a small fraction of the table
the field is a string, boolean, or numeric scalar

Good candidates include account_id, workspace_id, service, environment, event, and stable nested fields such as request.region.

For an agent that investigates incidents account by account, account_id is often a strong first column:

SELECT timestamp_utc, event, message
FROM app_events
WHERE account_id = 'acct_123'
  AND timestamp_utc >= now() - INTERVAL '24 hours'
ORDER BY timestamp_utc DESC

The time predicate still narrows the time window; the account predicate additionally lets Telemetry prune partitioned parts inside that window.

Put the most useful column first

Column order defines a leading-prefix hierarchy. Given:

{
  "partitionColumns": ["account_id", "event"]
}

Telemetry can use partition pruning for queries that filter by:

account_id
account_id and event

It cannot use this layout for a query that filters only by event, because the first column is missing. Choose the first column for the broadest set of important queries, then add another only when queries commonly filter by both.

Start with one column. Add a second only after you can describe the query pattern it improves. Long column lists create increasingly narrow write and compaction groups, while later columns help fewer queries.

Filters that enable pruning

Telemetry currently derives partition pruning from literal equality, positive IN, and IS NULL predicates joined with AND:

WHERE account_id = 'acct_123'

WHERE account_id IN ('acct_123', 'acct_456')

WHERE account_id = 'acct_123'
  AND event IN ('request_failed', 'request_retried')

WHERE account_id IS NULL

Range predicates, negation, and ambiguous OR expressions do not produce partition pruning. For example, partitioning on duration_ms does not help duration_ms > 1000. Queries are still correct; they simply scan without that optimization.

Nested scalar paths work too:

{
  "partitionColumns": ["request.region"]
}

SELECT count(*)
FROM app_events
WHERE request.region = 'us-west-2'

Lists, objects, and timestamps are not supported as partition columns. Timestamp filtering already has its own pruning path and is usually better expressed as a query time range.

What to avoid

Avoid choosing a field merely because it exists in every event. A useful partition column reflects a frequent, selective query.

Be cautious with:

fields almost every query ignores
fields queried mostly with ranges, LIKE, or negation
rapidly changing or unbounded values when queries rarely repeat them
multiple columns whose leading prefix does not match real query patterns

A low-cardinality field can still be useful when it removes a large portion of the table—for example, environment = 'production'. A high-cardinality identifier can be excellent for point investigations, but it may fragment writes and compaction. Prefer the field that removes the most data across the queries you actually care about, not simply the field with the most distinct values.

Rollout workflow

Inspect the queries run by people, dashboards, alerts, and agents. Identify the equality filter shared by the most expensive selective queries.
Confirm the field and type with GET /tables/<table>/schema.
Configure one partition column through the table settings UI or the Tables API.
Keep the important queries' time filters; partition pruning complements time pruning rather than replacing it.
Re-run representative queries after existing data has had time to be rewritten. Compare scanned work and latency, not just one warm-cache execution.
Add a second column only if common queries filter on the first and second columns together.

New data uses a changed partition specification immediately. Existing parts are rewritten in the background, and unrewritten parts remain queryable, so results stay complete during migration. Performance improves progressively as more of the table adopts the new layout.

Configure through the API

curl -X PATCH https://api.telemetry.sh/tables/app_events/partition-columns \
  -H "Content-Type: application/json" \
  -H "Authorization: $API_KEY" \
  -d '{
    "partitionColumns": ["account_id", "event"]
  }'

To change the order, send the complete desired list. To turn partitioning off, send null or an empty array:

curl -X PATCH https://api.telemetry.sh/tables/app_events/partition-columns \
  -H "Content-Type: application/json" \
  -H "Authorization: $API_KEY" \
  -d '{"partitionColumns": null}'

The API validates the fields against the table's current schema. See the Tables API reference for the exact request and response contract.

Use this doc with your coding agent