On-the-fly health checks

Tinybird is built around the idea of data that changes or grows continuously. This guide will show you how to use the built-in tools to monitor your data ingestion processes.

Once you have fixed all the possible errors in your source CSV files, matched the Data Source schema to your needs and done the on-the-fly transformations (if needed), at some point you'll start ingesting data periodically, and knowing the status of the your ingestion process will be key. Whether there are new rows in quarantine, or the jobs are failing recurrently or there is any other type of problem.

The Data Sources Log shows you all operations performed over your data for a given period of time

Apart from the tools we provide in our User Interface, here are couple of powerful tools you can use for advanced monitoring.

About Ingestion Jobs

Every time you run an ingestion process, from the UI, via the API or the CLI you generate what we call an Ingestion Job. If using the CLI or the Rest API, as a response you will get the Ingest Job status.

When wanting to programmatically check the status of the ingestion process you should keep requesting the {% code-line %}job_id{% code-line-end %} information until it's {% code-line %}done{% code-line-end %} or {% code-line %}error{% code-line-end %}.

It can be done with a simple request as follows:

Continuous monitoring using the Service Data Sources

Besides the Quarantine Data Sources, and the Ingestion Jobs, Service Data Sources will also help you with your ingestion health checks. They can be used like any other Data Source in your account, so you can create API endpoints to monitor your ingestion processes.

Although you can automatically create a Pipe from the Operations Log of any of your Data Sources, you can also create an empty Pipe and query the {% code-line %}tinybird.datasources_ops_log{% code-line-end %} directly.

For instance, you could list your ingest processes during the last week with a simple query like this

calculate the percentage of quarantined rows for a given period of time

or monitor the average duration of your periodic ingestion processes for a given Data Source.

If you want to configure or build an external service which monitors these metrics, you just need to create an API endpoint and raise an alert when passing a threshold. When receiving an alert, you could always check the Quarantine Data Source or the Operations Log to see what's going on and fix your CSV files or ingestion process.