High-frequency Ingestion

Intermediate
In this guide, we'll learn how to ingest JSONs at thousands of requests per second to Tinybird using v0/events. Several events can be ingested in the same request.

{% tip-box title="Feature preview" %}High-frequency ingestion is currently a feature preview and it's considered experimental.{% tip-box-end %}

Setup: create the target Data Source

Firstly, you need to create an NDJSON Data Source. You can use the API, or simply drag & drop a file on the UI.

Even though you can add new columns later on, you have to upload an initial file. The Data Source will be created and ordered based upon those initial values.

As an example, upload this NDJSON file:

Ingest from the browser: JavaScript

Ingesting from the browser simply requires making a simple POST request like this. Ensure to put your own API token and change the name of the target Data Source to the one you would have created as explained above.

Remember that publishing your admin token on a public web is a security vulnerability. We highly recommend you to create a new Auth token that restrict access granularity.

Ingest from the backend: Python

Ingesting from the backend is as simple as from the browser. Here is a Python snippet. As before, you'll need to set your Auth token and Data Source name.

Ingest from the command line: curl

For completeness, here is the curl snippet that sends two events in the same request.

Adding new columns from the UI

As you add extra information in the form of new JSON fields, the UI will prompt you to include those new columns on the Data Source.

As an example, we'll send a new event with an extra field:

After this, we can go to the UI's Data Source modal. We'll be asked if we want to add the new column.

Here, we'll be able to select the desired columns and adjust the types.

After we confirm the addition of the column, new events will populate it.

ON THIS GUIDE