Sending telemetry data to the Beep backend

beep
telemetry

#1

Dear @iconize and @vinz,

the commit Added dynamic sensor measurements input for backend · beepnl/BEEP@865aa97 · GitHub coming in two days ago sparked my interest as @caro told me she would like to forward measurement data from the Kotori data acquisition backend into the Beep backend. I assume both pieces of software are already up and running in the university’s data center in Bremen? Please correct me if I’m wrong.

In order to interconnect both systems with each other, having a “dynamic sensor measurement input” sounds just perfect. Would this enable us to just send over a bunch of data encapsulated in a flat dictionary or such?

I recognize there are some measurement field names already listed in BEEP/MeasurementSeeder.php at bob-additions · beepnl/BEEP · GitHub. Would this be a kind of a reference schema for me?

I will be happy to hear back from you regarding more detailed information about possible ways for interfacing.

With kind regards,
Andreas.


#2

Hi @Andreas,

yes, Beep and Kotori are currently running on our server at Uni Bremen.
Both, Kotori and Beep, use InfluxDB as database for the measurements and differ only in the structure of the tables.
Currently Beep reads the data from a database called bee_data and a single table sensors. All entries have the fields time, value and the tags key, name. The key can be configured in the Beep frontend and has to be a unique string. The name of the sensor should be one of the field names from the second link in your post.
As far as I understood these field names correspond to the units/labels that are displayed on the frontend.

Kind regards,
Vincent


#3

Hi Andreas,

Indeed you can dynamically enter new sensors in the Beep back-end. By using the "abbreviation" field as the "value_name" connected to a "value", you can post your data to the Beep API by using a POST to [url]/api/sensors with a body like:

{"key":"sensor_key","value_name1":numeric_value1,"value_name2":numeric_value2}

Sensor values will be stored in InfluxDB with the "key" (sensor key) and "name" (value_name) attribute.


#4

Dear @vinz and @iconize,

apologies for getting back so late to this after initiating our discussion about the integration work some time ago already. I was too optimistic to be able to start the work on that back then, but different things have been able to acquire more attention, some of them might be reflected in the forum while some other things have been happening beyond Hiveeyes and BOB.

Introduction

Finally: Thank you so much for your valuable information and good to hear that

As I already had imagined, we would have two ways to integrate the Kotori DAQ system with the Unified Beep system, right?

Alternatives

  1. Make Kotori store the measurement data to an InfluxDB database structure like implemented by Beep to enable Beep directly reading from there as outlined by @vinz.
  2. Make Kotori forward the measurement data to the designated Beep HTTP data acquisition endpoint as pointed out by @iconize.

Reasoning

The most prominent question which comes to mind: Would you prefer one way or the other to integrate both systems with each other? While saying that, these are my thoughts:

  • Personally, I will be happy to implement both ways.
  • The second variant (»Make Kotori forward the data to Beep«) would store the data redundantly, so we will have to account for doubling of the storage size. However, as the discrete telemetry data isn’t very dense compared to audio storage, the additional storage cost will probably be neglectable. On the other hand, this would make both systems run independently from each other – the integration would be loosely coupled then. Personally, I would consider this a good thing.
  • Besides the basic question, there are obviously more nitty gritty details to probably care about, like how to address multi-tenancy, authentication and associated things.
  • I’ve also heard from @caro that you might have been able to integrate both systems already – or did I get that detail wrong?

Bottom line

As I’m a more talkative guy and speech is way more dense than written text, I would like to be able to give both of you a call during the next days. Would you be available for that?

Otherwise, please let me know whether you might have solved all things already or if there are other or leftover things you would like to have addressed by me.

Thanks already and with kind regards,
Andreas.


#7

Hallo @caro,

mit @vinz haben wir besprochen, dass der Weg, die Daten optional/wahlweise/parallel direkt vom Meßknoten aus an beide Systeme zu senden, einfacher ist, anstatt die Systeme im Backend miteinander zu verbinden.

Wir würden den entsprechenden Code zur dynamischen Übermittlung der Meßdaten an das Beep System dann kurzerhand zur Telemetrieeinheit hinzufügen, um damit eine weitere Übertragungsvariante zu implementieren.

Wir hoffen das ist genau so im Interesse von BOB, da wir dieses Thema damit pragmatisch lösen können. Schließlich gibt es anderswo noch genug andere Baustellen auf dem Weg.

Viele Grüße,
Andreas.


#9

Hi @caro,

ich wollte hier nochmal nachhaken, ob das so wie oben angesprochen in Eurem Sinne ist. Für das nächste Release der Firmware wäre dann auch der Beep Adapter vorgesehen.

Viele Grüße,
Andreas.


#12

Hi @Andreas, bei mir kam das über @vinz als beschlossene Sache an. Er hatte das genau so formuliert wie du: ihr habt darüber gesprochen und habt euch für jetzt auf diese pragmatische Lösung geeinigt. Für die Zukunft wünschen wir uns alle und arbeiten an einer “Beep Adapter”-Variante.