Sending telemetry data to different backends

Andreas · February 17, 2019, 3:24pm

the commit Added dynamic sensor measurements input for backend · beepnl/BEEP@865aa97 · GitHub coming in two days ago sparked my interest as @caro told me she would like to forward measurement data from the Kotori data acquisition backend into the Beep backend. I assume both pieces of software are already up and running in the university’s data center in Bremen? Please correct me if I’m wrong.

In order to interconnect both systems with each other, having a “dynamic sensor measurement input” sounds just perfect. Would this enable us to just send over a bunch of data encapsulated in a flat dictionary or such?

I recognize there are some measurement field names already listed in BEEP/MeasurementSeeder.php at bob-additions · beepnl/BEEP · GitHub. Would this be a kind of a reference schema for me?

I will be happy to hear back from you regarding more detailed information about possible ways for interfacing.

With kind regards,
Andreas.

vinz · February 18, 2019, 8:14am

Hi @Andreas,

yes, Beep and Kotori are currently running on our server at Uni Bremen.
Both, Kotori and Beep, use InfluxDB as database for the measurements and differ only in the structure of the tables.
Currently Beep reads the data from a database called bee_data and a single table sensors. All entries have the fields time, value and the tags key, name. The key can be configured in the Beep frontend and has to be a unique string. The name of the sensor should be one of the field names from the second link in your post.
As far as I understood these field names correspond to the units/labels that are displayed on the frontend.

Kind regards,
Vincent

iconize · February 19, 2019, 9:52am

Hi Andreas,

Indeed you can dynamically enter new sensors in the Beep back-end. By using the "abbreviation" field as the "value_name" connected to a "value", you can post your data to the Beep API by using a POST to [url]/api/sensors with a body like:

{"key":"sensor_key","value_name1":numeric_value1,"value_name2":numeric_value2}

Sensor values will be stored in InfluxDB with the "key" (sensor key) and "name" (value_name) attribute.

Andreas · March 14, 2019, 8:39pm

Dear @vinz and @iconize,

apologies for getting back so late to this after initiating our discussion about the integration work some time ago already. I was too optimistic to be able to start the work on that back then, but different things have been able to acquire more attention, some of them might be reflected in the forum while some other things have been happening beyond Hiveeyes and BOB.

Introduction

Finally: Thank you so much for your valuable information and good to hear that

As I already had imagined, we would have two ways to integrate the Kotori DAQ system with the Unified Beep system, right?

Alternatives

Make Kotori store the measurement data to an InfluxDB database structure like implemented by Beep to enable Beep directly reading from there as outlined by @vinz.
Make Kotori forward the measurement data to the designated Beep HTTP data acquisition endpoint as pointed out by @iconize.

Reasoning

The most prominent question which comes to mind: Would you prefer one way or the other to integrate both systems with each other? While saying that, these are my thoughts:

Personally, I will be happy to implement both ways.
The second variant (»Make Kotori forward the data to Beep«) would store the data redundantly, so we will have to account for doubling of the storage size. However, as the discrete telemetry data isn’t very dense compared to audio storage, the additional storage cost will probably be neglectable. On the other hand, this would make both systems run independently from each other – the integration would be loosely coupled then. Personally, I would consider this a good thing.
Besides the basic question, there are obviously more nitty gritty details to probably care about, like how to address multi-tenancy, authentication and associated things.
I’ve also heard from @caro that you might have been able to integrate both systems already – or did I get that detail wrong?

Bottom line

As I’m a more talkative guy and speech is way more dense than written text, I would like to be able to give both of you a call during the next days. Would you be available for that?

Otherwise, please let me know whether you might have solved all things already or if there are other or leftover things you would like to have addressed by me.

Thanks already and with kind regards,
Andreas.

Andreas · April 3, 2019, 6:59pm

Hallo @caro,

mit @vinz haben wir besprochen, dass der Weg, die Daten optional/wahlweise/parallel direkt vom Meßknoten aus an beide Systeme zu senden, einfacher ist, anstatt die Systeme im Backend miteinander zu verbinden.

Wir würden den entsprechenden Code zur dynamischen Übermittlung der Meßdaten an das Beep System dann kurzerhand zur Telemetrieeinheit hinzufügen, um damit eine weitere Übertragungsvariante zu implementieren.

Wir hoffen das ist genau so im Interesse von BOB, da wir dieses Thema damit pragmatisch lösen können. Schließlich gibt es anderswo noch genug andere Baustellen auf dem Weg.

Viele Grüße,
Andreas.

Andreas · April 12, 2019, 8:59am

Hi @caro,

ich wollte hier nochmal nachhaken, ob das so wie oben angesprochen in Eurem Sinne ist. Für das nächste Release der Firmware wäre dann auch der Beep Adapter vorgesehen.

Viele Grüße,
Andreas.

caro · April 15, 2019, 1:43pm

Hi @Andreas, bei mir kam das über @vinz als beschlossene Sache an. Er hatte das genau so formuliert wie du: ihr habt darüber gesprochen und habt euch für jetzt auf diese pragmatische Lösung geeinigt. Für die Zukunft wünschen wir uns alle und arbeiten an einer “Beep Adapter”-Variante.

clemens · May 23, 2019, 2:51pm

Die grafische Darstellung schaut aktuell so aus:

dashboard like

klassisches Liniendiagramm

Andreas · June 26, 2019, 4:53pm

Hi there,

we are happy to share some updates with all of you.

Summary

While the software for the

RaspberryPi-based datalogger [1] was able to put measurement data into CSV and publish it using MQTT to the Hiveeyes backend

and the

MicroPython-based datalogger [2] was able to put measurement data into CSV and publish it to the Beep platform,

the

also MicroPython-based datalogger software [3] we just recently tried to push forward is now capable to publish data both to the Hiveeyes backend using MQTT or HTTP as well as to the Beep platform using HTTP. However, it is still lacking the functionality to store or buffer data into locally attached storage using CSV or other serialization formats.

Configuration

The configuration settings look like that.

Beep telemetry

github.com

hiveeyes/hiveeyes-micropython-firmware/blob/0.5.1/settings.example-bob.py#L95-L106


# Beep telemetry: JSON over HTTP
{
    # Enable/disable this telemetry target.
    'enabled': False,


    # Define telemetry endpoint and address information.
    'endpoint': 'https://bee-observer.org/api/sensors',
    'topology': 'beep-bob',
    'data': {
        'key': '## BEEP_SENSOR_KEY ##',
    },
},

Hiveeyes telemetry

github.com

hiveeyes/hiveeyes-micropython-firmware/blob/0.5.1/settings.example-bob.py#L79-L93


# Hiveeyes telemetry: JSON over MQTT
{
    # Enable/disable this telemetry target.
    'enabled': False,


    # Define telemetry endpoint and address information.
    'endpoint': 'mqtt://swarm.hiveeyes.org',
    'topology': 'mqttkit',
    'address': {
        "realm": "hiveeyes",
        "network": "testdrive",
        "gateway": "area-42",
        "node": "node-99",
    },
},

Cheers,
Andreas.

[1] GitHub - Hiverize/Sensorbeuten: Code, um die Messdaten aus den Sensorbeuten zu loggen und zu verarbeiten.
[2] GitHub - Hiverize/FiPy
[3] GitHub - hiveeyes/hiveeyes-micropython-firmware: Hiveeyes MicroPython Datalogger - data logging for humans.

Ping @iconize, @marten.schoonman, @caro, @vinz, @Diren, @didilamken and @MKO.

Andreas · June 26, 2019, 5:08pm

Mapping sensor field names to BEEP field names

Dear Pim,

iconize:

Indeed you can dynamically enter new sensors in the BEEP back-end […], you can post your data to the BEEP API by using a POST to [url]/api/sensors with a body like:
{
  "key": "sensor_key",
  "value_name1": 42.42,
  "value_name2": 84.84
}

Thanks @iconize for implementing the dynamic sensor API for BEEP.

After figuring out at Kontinuierliche Verbesserungen des Terkin-Datenloggers (600er) - #13 by Andreas what the actual used telemetry field names should be, we mapped them into the configuration settings of the datalogger software like that:

github.com

hiveeyes/terkin-datalogger/blob/0.5.1/settings.example-bob.py#L181-L198


      
          # Map sensor field names to telemetry field names.
          # Right now, please adapt this according to your sensor configuration by
          # looking at the console output of the line
          # `[terkin.datalogger] INFO: Sensor data`. Thanks!
          #
          # Remark:
          # This will be replaced by runtime configuration through
          # HTTP API and captive portal.
          sensor_telemetry_map = {
              "_version": "1.0.0",
              "temperature.0x77.i2c:0": "t",
              "humidity.0x77.i2c:0": "h",
              "pressure.0x77.i2c:0": "p",
              "weight.0": "weight",
              "temperature.28ff641d8fdf18c1.onewire:0": "t_i_1",
              "temperature.28ff641d8fc3944f.onewire:0": "t_i_2",
              "system.temperature": "t_i_5",
          }

While this does not happen interactively there yet, we are looking forward to the next steps to conclude together with @vinz

However, the datalogger software is already pretty much usable on its own even with manual configuration, so this might spark your interest or from some people you gathered around BEEP in the meanwhile.

Cheers again,
Andreas.

Diren · June 28, 2019, 8:45am

Hi @Andreas and everybody,
thanks for your posts and your work! I’ll try to set it up on my FiPy during the next days. Maybe we can have a phone call about how to merge the repositories next week? I think @vinz is busy preparing for exams…
Cheers,
Diren

MKO · June 28, 2019, 4:08pm

Hi @Diren,
If @vinz have no time, i can try. I have worked a litle bit with Angular cli ,at last Time.
@Andreas think that Framework work on FiPy, too.
I think that would be dificult for me.
Because some funktions we need, that i have never used.
If @vinz say he can make in the next time, or he worket on.
I don’t try.
I do not wan’t to destroy his work.

Andreas · July 1, 2019, 3:46pm

Hi @Diren,

Appreciate it. Thanks also for all your hard work on making the whole Bee Observer thing possible.

Good luck ^[1]. We will be happy to help if there will be any obstacles. As it seems to have worked for @clemens, @MKO and @waggi already – even on Windows(!) – I am confident we can make it work on your machine as well. Which operating system are you running actually?

Sure. Looking forward.

Good luck @vinz and also kudos for all the hard work you have been putting into GitHub - Hiverize/FiPy and GitHub - Hiverize/Webinterface already.

With kind regards,
Andreas.

We just found the canonical README file does not guide newcomers appropriately yet. In order to make things more clear, I would like to point to these three pieces of documentation which might work for you when following them along. Otherwise, please don’t hesitate to ask.
↩︎