Streaming Ingestion

Streaming ingestion is the practice of writing events into an analytical database as they happen, one small batch at a time, so that queries can return results within seconds of the underlying activity instead of waiting on a nightly ETL window.

Batch ingestion treats data as something you collect, process, and load on a schedule (usually nightly). Streaming ingestion treats it as a continuous flow: events arrive from producers, get validated and grouped into micro-batches, and are written to columnar storage in seconds. The ingest path runs on its own infrastructure, separate from the readers, so high write volume never blocks the queries on the other side.

The architectural choices come from this constraint. Producers send events over HTTP, message queues like Kafka, or change-data-capture streams from operational databases. The ingest layer batches them just enough for the columnar engine to write efficiently, validates them against a schema, and applies back-pressure if downstream falls behind. Exactly-once semantics, idempotency, and dead-letter handling become part of the contract.

In Tinybird, streaming ingestion lands through the Events API for HTTP-borne data, and through connectors for Kafka, S3, and operational databases. The receiving end is always a data source.

Related terms

Where it shows up in Tinybird

Full reference in the docs
Tinybird wordmark