How do measurements flow from MQTT to Grafana?

Benutzung / Usage Datenakquise / Data acquisition
1

Hi all

Together with @Geraldine we are trying to send scale data to swarm.hiveeyes.org and display it with Grafana.
I think we suceeded in using MQTT mosquitto_pub and we are seeing data received by using
mosquitto_sub -h swarm.hiveeyes.org -p 1883 -t 'hiveeyes/testdrive/#' -v

However we did not find information about what this data becomes and how it flows through Kotori through a graphic display in Grafana.
We saw in this page that there is a ToDo for that part of the process. If this is still the case could you give us some hints on how it works and how we could address the question ?

1 Like
2

Dear @lorenzo and @Geraldine,

welcome - data-wise ;] - to our small community. We are happy you decided to share your measurements with the collaborative data collection system and the overall community and especially that you managed to do so already.

It looks like some readings already arrived on the testdrive dashboard at 14:22 and 14:36 CET:
https://swarm.hiveeyes.org/grafana/d/000000207/hiveeyes-testdrive-automatic?orgId=2&from=1522146869217&to=1522160396490

It must have been you, right? Cheers!

Thanks for asking! Kotori just receives the measurement data from MQTT and stores it into InfluxDB while maintaining an automatic dashboard in Grafana for a quick start. After that, you should derive a personal dashboard from this automatic generated one to customize the display to your requirements. You can have a look at the other dashboards what is possible there.

If you want to continue your endeavors, we will be happy to provide you with

  • Credentials for publishing measurement readings to a personal MQTT data acquisition channel to move forward from the “testdrive” channel.
  • Access credentials for editing your personal dashboard in Grafana.

We will also be happy to answer any further general or deeper technical questions, please don’t hesitate.

With kind regards,
Andreas.

P.S.: We are aware that our static documentation is not up-to-date and thorough enough at different places and want to apologize for that. That said, we are always looking for people who could volunteer on this.

3

Thank you for the hint Andreas,
We succeeded to make it work, either through bash command-line (the easiest part) and through the python file you wrote together with Richard Pobering (a bit more tricky).
We noticed that :

  • the realm/network “hiveeyes testdrive automatic” works well and displays data sent at quasi real-time.
  • it also generates new x/y graphics slots if the gateway/node are new
  • however I did some testing with “hiveeyes testdrive2 automatic” and it does not respond. That’s maybe due to the allowed rights?

We would be happy to get the credentials you suggest to be able to play a bit more deeply with it!

1 Like
4

Dear @lorenzo and @Geraldine,

Cool thing, congratulations! We are happy you found the resources you needed to get started and are also looking forward to receive some words about things we could improve, there are definitively many things to work on.

This is correct, the Mosquitto ACL for the topic "hiveeyes/testdrive/#" allows anonymous writes, while write access to all other topics is only allowed when authenticated.

Sure, sorry it took that long! We just sent the credentials for two accounts to each of you and @Geraldine in a private message, together with an unique user id you would use instead of "testdrive". You are free to choose the "gateway" and "node" parameters at your own desire and likewise are free to establish as many channels as you like for testing, research or production purposes.

We outlined a reasonable naming convention at Hiveeyes data acquisition » Addressing:

  • “gateway” is your gateway identifier, it designates a sensor node location. Choose anything you like, you might use this for giving names to different sites. ¹

  • “node” is your sensor node identifier. Choose anything you like. This usually gets transmitted from a sensor device, so it will probably identify an individual hive. ¹

… but it’s really up to you!

Please let us know if this information is useful for you and if you have succeeded with authenticated access to the MQTT bus. Otherwise, please don’t hesitate to get back to us, we are here to help.

Have fun!

With kind regards,
Andreas.

¹ While you are the owner of this namespace hierarchy, remember the labels for “gateway” and “node” making up a data transmission channel will be visible on the collaborative ether. So the best thing would be to assign kind of nicknames to your sites and nodes which don’t reveal their identity and location. This helps to stay completely anonymous and in control while still recognizing the channels by sensible names on your own behalf.

P.S.: We collected some instructions about transmitting data and will just add them here for completeness as we believe you made the first steps successfully already.

Some instructions are partly in German and we are aware the documentation in general might have some gaps ;]. At least, you can use the translation button below each post to translate it into the language you have defined in your personal profile.

5

Dear @lorenzo and @Geraldine,

We just wanted to let you know we currently had to turn off the “Instant Dashboard” feature while we are investigating an issue with it. We hope to restore this functionality soon.

However, data acquisition is working and you can continue as desired. There’s just a few more clicks to do before seeing your measurements flowing into Grafana.

With kind regards,
Andreas.

6

Hi Andreas,
I understand the constraints you explain below, and we can manage them without problem.

Also I was going to answer you telling that « I hope we are not ones responsible for the system problems you’re having » but after reading your message in the forum it seems that yes we are !

As you say, hope you can find a fix and get the system more robust.

By the way, I wrote this short guide to get started with sending data to Grafana.

Maybe you can reuse it somewhere…

I can also share it on the forum if this was the right way

Regards

Lorenzo

7

Dear Lorenzo,

Ok, cool.

No worries, the fault was definitively on our behalf and the reason was a straight programming error, however it has its subtleties.

Looks good, we are on it.

Cool thing! Please share it on the forum and if you don’t mind we will definitively take it as a starting point to improve on our handbook-style documentation. Thanks in advance!

Cheers,
Andreas.

8

Dear @lorenzo and @Geraldine,

we had the pleasure to make the data acquisition system more robust as outlined at Kotori release 0.21.1 and are also happy to announce another feature described at Neues Dashboard für imkerliche Daten / New dashboard for beekeepers.

In the meanwhile, @Geraldine seems to have managed to successfully submit measurement data using the personal accounts we created for you. At this time, the “Instant Dashboard” feature has already been enabled again so she was probably able to instantly see data submitted to the system. Kudos!

We wish you good luck bringing the sensor nodes into the field and look forward to any suggestions you might have for improving the system.

Cheers,
Andreas.

P.S.: We will also be happy if you can share some details and/or pictures about the sensor machinery you are using or intend to use. As you recognize this is really a thing which is very multisided across all members of our community, we are always curious about the individual solutions everybody is using at the sensor side.

9

Here is the short guide I wrote to get started with sending data to Grafana.
Although once you get it working it seems quite straightforward, it could be helpful to other fellows. How to send data to HiveEyes Grafana dashboard v02.pdf (233.0 KB)

Regards
Lorenzo

3 Likes
10

Dear Lorenzo,

thank you so much for writing this information down in such a concise manner. Very much appreciated!

With kind regards,
Andreas.