While already being here, we can think about how to improve the monitoring of our systems for future data loss events.
Monitoring in-flight data
In the past, we focused on the individual beekeeper and built sensor infrastructure for detecting data loss on individual data acquisition channels, see also:
This currently works by watching individual MQTT bus topics and deducing events from detected anomalies.
In general, we still lack good general visibility into the data acquisition system at many places. In order to overcome this, we want to start with improving the situation around isolated vs. federated data, see Datenmischwerk.
Monitoring the whole database for freshness
Let’s just start with a minimal spike suitable for monitoring InfluxDB databases for freshness of data. It should be capable to work as a sensor for Icinga2.
To minimize complexity for a quick solution, we will send HTTP requests to the Grafana API in order to query the InfluxDB database. By checking the JSON response, we can see whether there are measurements since a defined time in the past:
# Address of Grafana API endpoint datasource=https://luftdaten.getkotori.org/grafana/api/datasources/proxy/2/query # InfluxDB query query='SELECT * FROM earth_43_sensors WHERE time > now() - 5m LIMIT 1' # Is there any data for the given query? http $datasource db==luftdaten_testdrive q=="$query" | jq '.results.series != null' # The result false