Telemetry architecture

Telemetry validates incoming JSON, buffers accepted events, and stores them as Parquet files in S3. SQL queries combine the buffer with stored data, so you can query recent events before the next upload.

Data ingestion and buffering

A Rust service checks each incoming JSON event against its table's schema. It adds compatible events to a buffer. The buffer uploads to S3 after 15 minutes or 10,000 rows.

Real-time querying

Queries read both buffered events and data stored in S3 or cached on disk. Events do not have to wait for a buffer flush to appear in results.

Schema evolution

Telemetry adds new fields to the table schema as events arrive. Existing rows have no value for those fields. Adding a field and changing its type are different operations. See schema evolution for migration rules and examples.

Fault tolerance and buffer management

The service flushes its buffer during an orderly shutdown. A dead letter queue holds data that could not be processed or flushed for recovery. These recovery paths do not make every failure lossless. Applications should still define retry behavior and verify delivery for events they cannot afford to lose.

Compaction

Compaction merges small Parquet files into larger ones. This reduces the number of files a query must open and lets Parquet compress data across more rows.

Choosing files for a query

Telemetry parses SQL into an abstract syntax tree to identify tables, time ranges, and filters. It uses file metadata to select the files needed for the query and retrieve them from S3.

Disk caching and query routing

Telemetry caches files on disk by their content and routes a tenant's queries to a server likely to have those files. A cache hit avoids an S3 download. Query latency still depends on the data scanned, the query, and whether the cache contains the required files.

Query execution

Apache DataFusion executes SQL against the selected files and returns the results. See the DataFusion SQL reference for supported query patterns.