Create a Kafka connection using the CLI with `tb datasource create --kafka` for a guided process, or manually create .connection and .datasource files. Your credentials are stored securely as secrets. Tinybird supports multiple authentication methods including SASL for cloud providers like Confluent Cloud, Amazon MSK, and others.

Tinybird's Kafka connector is fully managed—you don't run anything. We handle consumer autoscaling, offset management, failure recovery, and high availability. Just provide your Kafka credentials and start ingesting.

Yes. Tinybird Local is a Docker container that replicates the full Tinybird environment, including Kafka connector support. Use `tb dev` for local development, run tests with `tb test`, and deploy to staging/production environments.

Tinybird's Kafka connector is serverless with automatic consumer autoscaling. There's no limit on topics or throughput—we scale based on your workload. The default message size limit is 10MB per message. You don't manage scale; we handle it automatically.

Tinybird uses throughput-based pricing. You pay based on the volume of data you ingest and query, so costs scale directly with your actual usage. Tinybird offers a free tier to get started, with paid plans that scale elastically based on throughput.

The Kafka connector is included in all plans, including the Free tier. With throughput-based pricing, you can start free and scale as your data volume grows. For high-throughput requirements, Enterprise plans offer dedicated infrastructure optimized for large-scale streaming workloads.

Tinybird supports JSON and Avro message formats. You can configure the format using KAFKA_VALUE_FORMAT and KAFKA_KEY_FORMAT settings in your datasource file. Compressed messages (gzip) are automatically decoded on ingestion.

Yes. Tinybird integrates with Confluent Schema Registry for decoding Avro messages. For JSON messages with schema registry, use the KAFKA_VALUE_FORMAT and KAFKA_KEY_FORMAT settings (requires CLI 5.14.0+). Configure your schema registry URL in the .connection file using KAFKA_SCHEMA_REGISTRY_URL.

Messages that fail to match your schema are automatically sent to a quarantine data source (similar to a dead letter queue). Quarantine preserves the original message plus error details, allowing you to inspect issues and replay data after fixing problems. Built-in circuit breakers and backpressure prevent cascading failures.

Use the FORWARD_QUERY instruction in your .datasource file to define how existing data transforms to your new schema. When you deploy, Tinybird applies schema changes with zero downtime—your data remains queryable throughout the migration.

Key limits: 10MB max message size, rate limits on API requests (HTTP 429 when exceeded), and 10-second default query timeout (configurable on paid plans). Messages exceeding limits are quarantined, not dropped.