We recommend using the JSON format for MQTT data transmission, especially for sending multiple readings at once. On the other hand, you can also send single readings in a discrete way.
Is CSV-over-MQTT an important thing to you? If so, we might consider supporting it. However, while offering it over HTTP, we actually don’t recommend it because the payload format is not self-contained so there’s always room for anomalies. The reason we introduced it the other day was especially for supporting the Open Hive sensor nodes until they offer to transmit data using MQTT/JSON in the future.
Please don’t hesitate to get back to us for further assistance and questions.
Just let us know if you are ready getting your own slot for publishing data. Despite what the documentation says, we recently turned on MQTT authentication:
So we will have to create an account for you, which is no big deal. The best option would be to do it sooner than later in order to use authenticated communication properly from the firmware side. However, the “testdrive” channel will always be open to anonymous users for testing purposes, so just go ahead in the meanwhile!
With kind regards,
Andreas.
P.S.: Also, don’t hesitate to ask for authentication credentials for Grafana: After handing them out to you, you will be ready to customize your own Dashboard.
Please also recognize the Grafana panel is not strongly coupled to the ingress data stream. It is just generated once for your convenience but not updated for all edge cases. So i would strongly suggest to get authenticated (which for i will send the credentials to you) to be able to play with your own Dashboards on top of the data you are transmitting.
Dear Andreas, thanks again for your kind and effective support. I will explain the reason for that…well, all that trials :-) in the following.
I just received one of the ESP8266 MCUs I ordered (while I’m still waiting for the Huzzah, the WeMoS D1 pro, etc.), which is a small chinese Amica ESP8266-based board feauring a CP2102 USB UART.
Then, with great enthusiasm, I cloned the Hiveeyes arduino repository and immediately built the node-wifi-mqtt. As the binary compiled from scratch makes the MCU to reboot, probably because no peripherals are connected to it, I started trimming the code, just to be able to send fake temperature values through Wi-Fi (i.e. temperature1.publish(myFixedCharString))
With respect to MQTT stuff, I left the AIO_USERNAME and AIO_KEY defines empty and left uncommented just the following line:
In which I replaced hiveeyes*airtemperature with /hiveeyes/testdrive/sardinia/node1 , according to the proposed addressing scheme, which also worked when using mosquito through my workstation.
As I wasn’t able to see any data to Grafana, and also I didn’t sniff what the MCU was sending with wireshark (I just put my phone in tethering mode, again for the enthusiasm :-), I performed some trial-and-error tests through the MCU and by console (along the trials, I was not only trying to modify the addressing scheme, but I was also trying to send (wrong) CSV data from my workstation, being not sure the MCU was adopting JSON).
I agree with your recommendations regarding JSON. I finally add I wasn’t able to get useful information by reading the Adafruit MQTT Library Documentation [1]
thanks for your persistence and for providing further information, especially that you are also going with the most simple firmware source code »node-wifi-mqtt« available from our repository. To be honest, it was never about to be released as Clemens is reflecting here (German, sorry):
Funny enough, @Thias is also currently looking at this, because he also wants to have a slow start getting more and more into the details gradually. When trying to use a more recent version of this source code, which is already doing JSON serialization, but does not live in the master branch yet, he found some errors on the telemetry side - sensor readings accidentally have been transmitted as strings, see also:
So i took the chance to reiterate on this once more: The current version of this firmware source code is living in the branch “node-wifi-mqtt-json” and has just been pushed to GitHub at node-wifi-mqtt.ino. I also tried to improve the inline documentation, especially regarding MQTT authentication: AIO_KEY is now gone.
As i personally don’t have any hardware in place, it still hasn’t been tested on any iron, but i would be happy if you and @Thias could take it out for a ride and provide feedback.
We will make it! Please keep up your enthusiasm!
Thanks,
Andreas.
P.S.: Please remember that the MQTT communication is about to be authenticated from now as already outlined above. We will provide authentication credentials and appropriate information in a few minutes.
Thanks again Andreas. Among my trials, I also invoked the publish method directly giving a float type as an argument, as performed in the Adafruit mqtt_esp8266 example [1]. I was wondering why to perform such a conversion, when I just noted you just updated the code avoiding the dtostrf call, thus saving power.
to maybe clarify things a bit more: The version in the “master” branch (node-wifi-mqtt.ino) is sending readings to the MQTT bus in a discrete way, i.e. one by one. The version in the branch “node-wifi-mqtt-json” (node-wifi-mqtt.ino [JSON]) however is already using JSON as a data serialization format to send all sensor readings en bloc, which we prefer.
Please note that with the former version it does not matter in which format the data is written to the MQTT bus: The discrete measurement values are actually going over the wire as strings and will be converted to floating point values before being written to the database. However, with the latter version using JSON, it is important to put them into the data container as floating point already.
Additionally, the former (non-JSON) version is running in production on @karsten’s ESP8266 node (see Welcome, Karsten!), while the latter version using JSON is still untested yet.
Thanks Andreas,
today I cloned the node-wifi-mqtt-json branch and started modifying the code to send fake values. In fact, the MCU was crashing once sending the JSON payload. Thus, after some debug I found I was unable to send a so long payload. I then tried sending less values, and also modified the code in order to send a better tailored string as follows:
int len = root.measureLength();
char payload[len];
root.printTo(payload, len);
Now, either using my custom MQTT topic [ “hiveeyes/2cdd923f-751c-4844-86a5-b9e868458441/sardinia/node1/data.json” ] or the testdrive automatic one, even if the mqtt_publisher.publish method returns success, However, I’m not able to see data within grafana. I still didn’t sniff data through wireshark. Can you please check if the hiveeyes broker received any data?
The message seems to lack one single character to be a valid JSON payload, so doing char payload[len+1]; could probably help. Thanks for mentioning root.measureLength(), i wasn’t aware that this is available as a method on the root object.
I’m happy we’re making progress on this! So i can assume the basics as well as the MQTT authentication already works using this firmware? Great, then i’m inclined to merge this to the master branch as soon as possible.
With kind regards,
Andreas.
P.S.: Sorry we don’t have any error signalling in place yet. The error message produced by Kotori when receiving your payload actually was:
button just below the post.