Ingest AWS ELB logs¶

AWS Elastic Load Balancer (ELB) logs record detailed information about requests sent to your load balancers. These logs are useful for monitoring traffic, troubleshooting, and analyzing application performance.

What are AWS ELB logs?¶

AWS ELB logs capture information such as the time a request was received, the client's IP address, request paths, backend responses, and more. Each log entry is a single line in a space-delimited text format.

Here's an example of an ELB log entry:

2023-06-01T12:00:00.000000Z my-elb 192.0.2.1:12345 203.0.113.1:80 0.000022 0.001048 0.00002 200 200 0 57 "GET http://www.example.com:80/ HTTP/1.1" "curl/7.46.0" - -

For a full description of each field, see the AWS documentation.

File format¶

AWS ELB logs are typically exported to Amazon S3 as compressed files with the .log.gz extension. Each file contains multiple log entries, one per line.

• Format: Space-delimited text
• Compression: Gzip (.gz)

Importing ELB logs into Tinybird¶

You can import ELB logs into Tinybird using the Data Sources API with format=csv and a custom delimiter. Although ELB logs are not comma-separated, you can specify the space character as the delimiter.

Step 1: Prepare your Data Source schema¶

Define a schema that matches the ELB log fields. For example:

SCHEMA >
  `timestamp` DateTime,
  `elb` String,
  `client_port` String,
  `backend_port` String,
  `request_processing_time` Float32,
  `backend_processing_time` Float32,
  `response_processing_time` Float32,
  `elb_status_code` UInt16,
  `backend_status_code` UInt16,
  `received_bytes` UInt64,
  `sent_bytes` UInt64,
  `request` String,
  `user_agent` String,
  `ssl_cipher` String,
  `ssl_protocol` String
ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"

Adjust the schema to match your log format and needs.

Step 2: Upload and ingest the logs¶

You can send your .log.gz files directly to Tinybird. Tinybird will automatically decompress and parse the file as CSV, as long as you set format=csv in the request. By default, Tinybird expects comma-separated values. Since ELB logs are space-delimited, set the delimiter explicitly with the dialect_delimiter parameter:

curl \
  -H "Authorization: Bearer <your_auth_token>" \
  -X POST "https://api.tinybird.co/v0/datasources?name=logs&format=csv&dialect_delimiter=%20" \
  -F "csv=@file.log.gz"

Next steps¶

• Data Sources API reference
• AWS ELB access log documentation