PostgreSQL CDC with Redpanda Connect

Redpanda Connect is an ecosystem of high-performance streaming connectors that serves as a simplified and powerful alternative to Kafka Connect.

Tinybird is the ideal complement to Postgres for handling OLAP workloads. The following guide shows you how to use Redpanda Connect to ingest data from a Postgres database into Tinybird.

Before you start

Before you connect Postgres to Redpanda, ensure:

  • You have a Redpanda cluster and Redpanda Connect installed with version 4.43.0 or higher. The following instructions use Redpanda Serverless, but you can use Redpanda Cloud Dedicated or self-hosted.
  • You have a PostgreSQL database with logical replication enabled.

Connect Postgres to Redpanda

  1. In the Redpanda Cloud console, select Connect from the navigation menu, then select Create pipeline.

  2. Add the pipeline configuration. You need the following information:

  • Postgres connection string (dsn)
  • Redpanda brokers (seed_brokers)
  • SASL mechanism (mechanism)
  • Username (username)
  • Password (password)

Use the following YAML template:

input:
  label: "postgres_cdc"
  postgres_cdc:
    dsn: <<postgresql://user:pass@host:port/db>> 
    include_transaction_markers: false
    slot_name: test_slot_native_decoder
    snapshot_batch_size: 100000
    stream_snapshot: true
    temporary_slot: true
    schema: public
    tables:
      - <<Table name>>

output:
 redpanda:
   seed_brokers:
     - ${REDPANDA_BROKERS}
   topic: <<Topic name>>
   tls:
     enabled: false
   sasl:
     - mechanism: SCRAM-SHA-512
       password: <<Password>>
       username: <<Username>>

See the Redpanda Connect docs for more information on the redpanda output and postgres_cdc input.

  1. Start the Redpanda Connect pipeline

Select Create to save and create the pipeline. This takes you back to the pipeline screen, where you can find your new pipeline. Open the new pipeline to view the logs and confirm that the pipeline is running.

Select the Topics page from the navigation menu and confirm that the topic exists and that messages are being produced.

  1. Connect Redpanda to Tinybird

Create a new Kafka connection and data source. See Kafka for more information.

Redpanda Connect continuosly consumes changes from Postgres and pushes them to your Redpanda topic. Tinybird consumes the changes from Redpanda in real time, making them available to query with minimal latency.

See also

Updated