Problem with MQTT/JSON transmission using the "node-wifi-mqtt" firmware

Dear @gtuveri,

by chance we were just running

mosquitto_sub -h swarm.hiveeyes.org -t '#' -v

where we recognized some attempts to send CSV data, most probably from you:

hiveeyes/testdrive/sardinia/node1/temp1/data.csv temperature
hiveeyes/testdrive/sardinia/node1/temp1/data.csv 10.0
hiveeyes/testdrive/sardinia/node1/temp1/data.csv temperature, 10.0
hiveeyes/testdrive/sardinia/node1/temp1/data/temperature.csv 10.0

Unfortunately, we don’t offer the CSV data serialization format over MQTT yet. Currently, you can transmit CSV data using HTTP only.

We recommend using the JSON format for MQTT data transmission, especially for sending multiple readings at once. On the other hand, you can also send single readings in a discrete way.

Is CSV-over-MQTT an important thing to you? If so, we might consider supporting it. However, while offering it over HTTP, we actually don’t recommend it because the payload format is not self-contained so there’s always room for anomalies. The reason we introduced it the other day was especially for supporting the Open Hive sensor nodes until they offer to transmit data using MQTT/JSON in the future.

Please don’t hesitate to get back to us for further assistance and questions.

With kind regards,
Andreas.

I see you are now using the JSON format:

hiveeyes/testdrive/sardinia/node1/temp1/data.json {"temperature": 12.3}

So things are turning out well:
https://swarm.hiveeyes.org/grafana/dashboard/db/hiveeyes-testdrive-automatic

Great!


Just let us know if you are ready getting your own slot for publishing data. Despite what the documentation says, we recently turned on MQTT authentication:

So we will have to create an account for you, which is no big deal. The best option would be to do it sooner than later in order to use authenticated communication properly from the firmware side. However, the “testdrive” channel will always be open to anonymous users for testing purposes, so just go ahead in the meanwhile!

With kind regards,
Andreas.

P.S.: Also, don’t hesitate to ask for authentication credentials for Grafana: After handing them out to you, you will be ready to customize your own Dashboard.

Dear @gtuveri,

Actually, this is also not fully correct. There’s one identifier too much inside the topic. It should just be:

hiveeyes/testdrive/sardinia/node1/data.json {"temperature": 12.3}

Speaking about the current addressing scheme, this would mean:

realm:   hiveeyes
network: testdrive
gateway: sardinia
node:    node1

Please also recognize the Grafana panel is not strongly coupled to the ingress data stream. It is just generated once for your convenience but not updated for all edge cases. So i would strongly suggest to get authenticated (which for i will send the credentials to you) to be able to play with your own Dashboards on top of the data you are transmitting.

Keep it up!

Cheers,
Andreas.

Dear Andreas, thanks again for your kind and effective support. I will explain the reason for that…well, all that trials :-) in the following.
I just received one of the ESP8266 MCUs I ordered (while I’m still waiting for the Huzzah, the WeMoS D1 pro, etc.), which is a small chinese Amica ESP8266-based board feauring a CP2102 USB UART.
Then, with great enthusiasm, I cloned the Hiveeyes arduino repository and immediately built the node-wifi-mqtt. As the binary compiled from scratch makes the MCU to reboot, probably because no peripherals are connected to it, I started trimming the code, just to be able to send fake temperature values through Wi-Fi (i.e. temperature1.publish(myFixedCharString))

With respect to MQTT stuff, I left the AIO_USERNAME and AIO_KEY defines empty and left uncommented just the following line:

Adafruit_MQTT_Publish temperature1 = Adafruit_MQTT_Publish(&mqtt, AIO_USERNAME “hiveeyes/kh/cfb/hive1/measure/airtemperature”);

In which I replaced hiveeyes*airtemperature with /hiveeyes/testdrive/sardinia/node1 , according to the proposed addressing scheme, which also worked when using mosquito through my workstation.
As I wasn’t able to see any data to Grafana, and also I didn’t sniff what the MCU was sending with wireshark (I just put my phone in tethering mode, again for the enthusiasm :-), I performed some trial-and-error tests through the MCU and by console (along the trials, I was not only trying to modify the addressing scheme, but I was also trying to send (wrong) CSV data from my workstation, being not sure the MCU was adopting JSON).
I agree with your recommendations regarding JSON. I finally add I wasn’t able to get useful information by reading the Adafruit MQTT Library Documentation [1]

[1] GitHub - adafruit/Adafruit_MQTT_Library: Arduino library for MQTT support

Dear Giuseppe,

thanks for your persistence and for providing further information, especially that you are also going with the most simple firmware source code »node-wifi-mqtt« available from our repository. To be honest, it was never about to be released as Clemens is reflecting here (German, sorry):

Funny enough, @Thias is also currently looking at this, because he also wants to have a slow start getting more and more into the details gradually. When trying to use a more recent version of this source code, which is already doing JSON serialization, but does not live in the master branch yet, he found some errors on the telemetry side - sensor readings accidentally have been transmitted as strings, see also:

So i took the chance to reiterate on this once more: The current version of this firmware source code is living in the branch “node-wifi-mqtt-json” and has just been pushed to GitHub at node-wifi-mqtt.ino. I also tried to improve the inline documentation, especially regarding MQTT authentication: AIO_KEY is now gone.

As i personally don’t have any hardware in place, it still hasn’t been tested on any iron, but i would be happy if you and @Thias could take it out for a ride and provide feedback.

We will make it! Please keep up your enthusiasm!

Thanks,
Andreas.

P.S.: Please remember that the MQTT communication is about to be authenticated from now as already outlined above. We will provide authentication credentials and appropriate information in a few minutes.

1 Like

Thanks again Andreas. Among my trials, I also invoked the publish method directly giving a float type as an argument, as performed in the Adafruit mqtt_esp8266 example [1]. I was wondering why to perform such a conversion, when I just noted you just updated the code avoiding the dtostrf call, thus saving power.

[1] Adafruit_MQTT_Library/mqtt_esp8266.ino at master · adafruit/Adafruit_MQTT_Library · GitHub

Dear Giuseppe,

to maybe clarify things a bit more: The version in the “master” branch (node-wifi-mqtt.ino) is sending readings to the MQTT bus in a discrete way, i.e. one by one. The version in the branch “node-wifi-mqtt-json” (node-wifi-mqtt.ino [JSON]) however is already using JSON as a data serialization format to send all sensor readings en bloc, which we prefer.

Please note that with the former version it does not matter in which format the data is written to the MQTT bus: The discrete measurement values are actually going over the wire as strings and will be converted to floating point values before being written to the database. However, with the latter version using JSON, it is important to put them into the data container as floating point already.

Additionally, the former (non-JSON) version is running in production on @karsten’s ESP8266 node (see Welcome, Karsten!), while the latter version using JSON is still untested yet.

However, @Thias will give it a try tomorrow, see:

Thanks for your patience,
Andreas.

Thanks Andreas,
today I cloned the node-wifi-mqtt-json branch and started modifying the code to send fake values. In fact, the MCU was crashing once sending the JSON payload. Thus, after some debug I found I was unable to send a so long payload. I then tried sending less values, and also modified the code in order to send a better tailored string as follows:

int len = root.measureLength();
char payload[len];
root.printTo(payload, len);

Now, either using my custom MQTT topic [ “hiveeyes/2cdd923f-751c-4844-86a5-b9e868458441/sardinia/node1/data.json” ] or the testdrive automatic one, even if the mqtt_publisher.publish method returns success, However, I’m not able to see data within grafana. I still didn’t sniff data through wireshark. Can you please check if the hiveeyes broker received any data?

Many thanks (and sorry!)

Giuseppe

1 Like

Hi Giuseppe,

you can subscribe to the MQTT bus as well. Just issue:

mosquitto_sub -h swarm.hiveeyes.org -t '#' -v

There, i’m seeing things like:

hiveeyes/2cdd923f-751c-4844-86a5-b9e868458441/sardinia/node1/data.json {"weight":100.00

The message seems to lack one single character to be a valid JSON payload, so doing char payload[len+1]; could probably help. Thanks for mentioning root.measureLength(), i wasn’t aware that this is available as a method on the root object.

I’m happy we’re making progress on this! So i can assume the basics as well as the MQTT authentication already works using this firmware? Great, then i’m inclined to merge this to the master branch as soon as possible.

With kind regards,
Andreas.

P.S.: Sorry we don’t have any error signalling in place yet. The error message produced by Kotori when receiving your payload actually was:

2017-03-31T11:56:25+0200 [kotori.daq.services.mig            ] ERROR:
Error processing MQTT message from topic "hiveeyes/2cdd923f-751c-4844-86a5-b9e868458441/sardinia/node1/data.json": 
[Failure instance: Traceback: <type 'exceptions.ValueError'>: Expecting object: line 1 column 16 (char 15)

    [...]
	/usr/lib/python2.7/threading.py:810:__bootstrap_inner
	/usr/lib/python2.7/threading.py:763:run
	/opt/kotori/lib/python2.7/site-packages/twisted/_threads/_threadworker.py:46:work
	/opt/kotori/lib/python2.7/site-packages/twisted/_threads/_team.py:190:doWork
	--- <exception caught here> ---
	/opt/kotori/lib/python2.7/site-packages/twisted/python/threadpool.py:250:inContext
	/opt/kotori/lib/python2.7/site-packages/twisted/python/threadpool.py:266:<lambda>
	/opt/kotori/lib/python2.7/site-packages/twisted/python/context.py:122:callWithContext
	/opt/kotori/lib/python2.7/site-packages/twisted/python/context.py:85:callWithContext
	/opt/kotori/lib/python2.7/site-packages/kotori/daq/services/mig.py:115:process_message
	/usr/lib/python2.7/json/__init__.py:338:loads
	/usr/lib/python2.7/json/decoder.py:366:decode
	/usr/lib/python2.7/json/decoder.py:382:raw_decode
	]

We already do have it on our agenda, but didn’t get around implementing it yet, see:

Sorry, it’s in German. But you can translate it by using the button just below the post.

Finally it worked! Great!

Giuseppe

Cool!
https://swarm.hiveeyes.org/grafana/dashboard/db/hiveeyes-2cdd923f-751c-4844-86a5-b9e868458441-automatic?from=1490957367488&to=1490957370184

Thanks for your patience and sorry for the rough edges we still have scattered around some places of the whole system :-)!

I’ve addressed this per:
https://github.com/hiveeyes/arduino/commit/9c86aa1a

Can i ask you to verify it?

Thanks in advance,
Andreas.