Ingest data using Vector.dev

Vector.dev is an open-source tool created by DataDog for collecting, transforming, and shipping logs, metrics, and traces.

Some common use cases for using Vector.dev as a Tinybird sink include:

  1. Ingesting data from a number of Vector.dev sources to Tinybird.
  2. Enriching other data sources with real-time Vector metrics.
  3. Aggregate logs and metrics from Vector.dev to Tinybird.
  4. Transform and redact sensitive data before ingesting it into Tinybird.

Read on to learn how to use the Events API as a Vector.dev sink.

Before you start

Before you connect Vector.dev to Tinybird, ensure:

  • You have installed Vector.dev.
  • You have a Tinybird workspace.

Use the Events API as a Vector.dev sink

To push events to Tinybird from Vector.dev, you need to configure Vector to use the Events API as a sink.

You can use the following example Vector configuration to push events, in this case Docker logs, to Tinybird:

sources:
  docker_logs:
    type: "docker_logs"

transforms:
  remap_docker_logs:
    inputs:
      - "docker_logs"
    type: "remap"
    source: |
      . = parse_json!(.log)

sinks:
  push_docker_logs_to_tinybird:
    inputs:
      - "remap_docker_logs"
    type: "http"
    uri: "$TINYBIRD_HOST/v0/events?name=docker"
    auth:
      strategy: "bearer"
      token: "$TINYBIRD_TOKEN"
    encoding:
      codec: "json"
    framing:
        method: "newline_delimited"

The previous snippet uses the docker_logs source to collect Docker logs, and the remap_docker_logs transform to parse the logs as JSON.

The push_docker_logs_to_tinybird sink uses the Events API to push the transformed logs to Tinybird in NDJSON (newline_delimited) format.

You can customize the $TINYBIRD_HOST and $TINYBIRD_TOKEN environment variables to use your Tinybird workspace.

Learn more about other sources you can use to ingest data into Tinybird in the Vector.dev documentation.

Updated