MQTT payload fields from TTN not working atm

Andreas · November 21, 2019, 8:06am

@Thias told us about recent issues with TTN when subscribing to one of their MQTT topics. As a recap, he is Using the Adafruit Feather M0 LoRa (RFM95) and TTN and submits measurement data using MQTT Weiterleitung per Bridge Konfiguration an swarm.hiveeyes.org.

Regarding this, we also found:

MQTT used to work fine, but stopped getting data - Application Development - The Things Network
Changes in the backend: Much better now! - TTN Backend - The Things Network
No data on MQTT when deploying TTN v3.2.6 - The Things Stack v3 - The Things Network

@Thias said it’s only about the /* suffix which has been disabled. Before, he was getting the discrete payloads containing just a flat dictionary of decoded CayenneLPP fields. It was easy to rewrite the MQTT topic address using an appropriate Mosquitto bridge configuration snippet to make Kotori ingest the measurement data.

Now, we will have to get a little bit smarter. In order to mitigate this, we might think about finally getting Kotori ingest the full TTN payload by adding an appropriate decoder in the spirit of More data acquisition payload formats for Kotori, see also TTN-Daten an Kotori weiterleiten.

Thias · November 22, 2019, 11:29am

From the TTN Slack #ops channel:
TTN_MQTT_fields_off

Andreas · November 23, 2019, 2:27am

Dear @Thias,

while the decoder is no rocket science at all, we haven’t made it to integrate it into Kotori in time. Now, I will be traveling until 7th of December or so and would like to apologize that this means some data loss for your hive monitoring setup.

The decoder is already half-finished within the ttnlogger program

so you might be able to pick it up from there and put it into something which republishes the telemetry data to the MQTT bus on swarm.hiveeyes.org in the appropriate format. However, you will probably have to put some hours of work into this guy to make that happen.

If you will really get into it, you might want to run it on a local RaspberryPi machine or even on elbanco itself within a screen or tmux session as a provisional solution.

Sorry again and with kind regards,
Andreas.

Appendix

These are pointers to the appropriate functions of the TTNDatenpumpe and InfluxDatabase classes within ttnlogger.

github.com

daq-tools/ttnlogger/blob/master/ttnlogger/ttn_logger.py#L22-L27


    self.split = split
    self.nogeo = nogeo

def on_receive(self, message=None, client=None):
    #print('on_receive:', message, client)

github.com

daq-tools/ttnlogger/blob/master/ttnlogger/ttn_logger.py#L77-L113


    def disconnect_callback(self, res, client):
        address = client._MQTTClient__mqtt_address
        print('Disconnected from MQTT broker at "{}"'.format(address))
        self.connected = False

    def uplink_callback(self, msg, client):
        print('-' * 63)
        print('Received uplink message from device "{}"'.format(msg.dev_id))
        if self.receive_callback is not None:
            self.receive_callback(message=msg, client=client)


class InfluxDatabase:

    def __init__(self, database='ttnlogger', measurement='data', nogeo=False):

        assert database, 'Database name missing or empty'
        assert measurement, 'Measurement name missing or empty'

        self.database = database

This file has been truncated. show original

Andreas · November 23, 2019, 6:14am

Dear @Thias,

we’ve just installed ttnlogger on elbanco and this might help you in the interim. The documentation about how to invoke this utility can be found within the README section Synopsis of ttnlogger.

It will be easy to figure it out how to invoke ttnlogger. I would recommend using a test database/measurement first and – as soon you can confirm everything is right – you might want to consider switching the output to your personal database/measurement.

In order to keep it running after logout, you might want to invoke it within a tmux session like

tmux new -s ttn-thias ttnlogger testdrive ttn-account-v2.UcOZ3_gRRVbzsJ1lR7WfuINLN_DKIlc9oKvgukHPGck testdb data

With kind regards,
Andreas.

Thias · November 25, 2019, 1:23pm

Thanks! It’s basically working after a (Python 2->3 ?) bugfix.

It’s running right now in a screen environment on elbanco and data is visible in a dashboard duplicated from my original live data dashboard.

Next thing would be to get rid of the database and measurement arguments in favor of deriving both from the dev_id. I’ll see what I can do but I don’t expect to get this implemented by myself properly since some structural rewritings are needed.

Enjoy your holidays!

Thias · November 25, 2019, 10:41pm

I accidentally got it working with dynamic InfluxDB database and measurement assignments which are being derived from the dev_id. The PR is extended accordingly

The code on elbanco is now running on a separate branch which can be deleted before pulling my commits to master.

With this change implemented the TTN dev_ids will have to follow a certain scheme:
A dev_id: hiveeyes-USER-LOCATION-NAMEOFHIVE will be written to the agents database hiveeyes-USER while the measurement will be LOCATION-NAMEOFHIVE, where you want to replace upper case strings with your individual (lower case) namings.

ToDo: adapt documentation [edit: done]

Thias · November 28, 2019, 9:26pm

PR was again extended by a commit to store signal quality data from all receiving gateways.

limago · December 1, 2019, 7:03am

Here the official Status update from TTN:

For performance reasons we unfortunately had to disable publishing Uplink Fields to MQTT in our EU region. This means that the values from payload decodes are no longer published to individual [AppID]/devices/[DevID]/up/[Field] topics. Until this is resolved, you can subscribe to [AppID]/devices/[DevID]/up and read the payload_fields field from the uplinks you receive there.
Posted 13 days ago. Nov 18, 2019 - 11:00 CET