Terkin for MicroPython

As @clemens is already experimenting with LoRa using the …

… see also …

… we stepped in and provided convenient telemetry client wrapper libraries for MicroPython to receive data from LoRa and forward it to MQTT or HTTP.

Demo example

Besides others, there’s a simple demo program which sends dummy telemetry data in CSV format to the HTTP endpoint of “swarm.hiveeyes.org”:

As you can see here, this already does the job on MicroPython for Unix:

Transmit data

$ micropython -c 'import sys; sys.path.insert(0, ".")' examples/node_http_csv_raw.py

Channel URI:  http://swarm.hiveeyes.org/api-notls/hiveeyes/testdrive/area-42/node-1
HTTP Path:    /api-notls/hiveeyes/testdrive/area-42/node-1/data
Payload:      ## temperature, humidity, weight

Channel URI:  http://swarm.hiveeyes.org/api-notls/hiveeyes/testdrive/area-42/node-1
HTTP Path:    /api-notls/hiveeyes/testdrive/area-42/node-1/data
Payload:      42.42, 84.84, 35

Receive data

After receiving the data on the HTTP interface, Kotori republishes it to the MQTT bus, so:

$ mosquitto_sub -h swarm.hiveeyes.org -p 1883 -t 'hiveeyes/testdrive/#' -v
hiveeyes/testdrive/area-42/node-1/data.json {"temperature": 42.42, "humidity": 84.84, "weight": 35.0}

LoRa-to-HTTP gateway example

There’s also a gateway/forwarder which receives data from the LoRa subsystem and sends it to HTTP as well. Everything is forwarded verbatim without any decoding involved, the payload just is being announced as “Content-Type: text/csv” when performing the HTTP POST request:

It is based on the contribution “lopy_lora_server.py” by @constantinos on the Pycom Forum (thanks!):


Note: Please be aware that this hasn’t been run on any real iron yet. We appreciate your help in doing this!

Hi there,

we are happy to announce that we have been able to take this to the next level. Based on good experiences by @Ron, @tonke, @RalfL and others with MicroPython (thanks for sharing your efforts!), we are currently working on GitHub - hiveeyes/hiveeyes-micropython-firmware: Hiveeyes MicroPython Datalogger -- Data logging for humans.

After resolving some dependency hell aspects (e.g. How to use "libpcre" on the Pycom MicroPython runtime? | Pycom user forum) - which is just business as usual - we are now running the telemetry subsystem of the Hiveeyes MicroPython firmware on real hardware (the FiPy by Pycom) and it works pretty well already.

The foundation of the configuration system has been laid out recently (see settings.example.py), so we are tagging this with 0.1.0 now to signal official alpha quality.

@einsiedlerkrebs is already working on LoRaWAN/TTN telemetry while we are also taking considerations into account which have been discussed with @tonke recently. Thank you so much!

As all of this is in a very early state yet, we are looking forward to get more eyeballs to it. Please ask us anything, we will be happy to give any answers and appropriate onboarding assistance.

With kind regards,
Andreas.


References

Bildernotiz mit Zeitstempel. Alles auch auf dachboden-testdrive-fipy.

Automatic telemetry field names

About

image

TTN-native telemetry

About

image


Ziel: Beides ineinander konvergieren lassen ;].

1 Like

Status quo

Introduction

Together with @clemens, we have been able to confirm that code from the current development state of the Hiveeyes MicroPython Firmware works successfully with and without sensors attached and MQTT-over-WiFi telemetry enabled.

image

Pre-flight

Upgrade the Pycom device firmware to a recent version, see hiveeyes-micropython-firmware/pycom-firmware-upgrade.rst at master · hiveeyes/hiveeyes-micropython-firmware · GitHub.

Synopsis

# Find serial port.
make list-serials

# Configure serial port.
export MCU_PORT=/dev/ttyS16

# Pre-flight checks.
make list-boards

# Bootstrap dependencies once.
# Usage: After updating the list of dependencies, e.g. through "git pull".
make setup

# Upload firmware framework and sketch and trigger a cold start.
# Usage: Each time you modify code.
make recycle
1 Like

Plans for MicroTerkin 0.3.0

The Hiveeyes MicroPython Firmware release 0.3.0 is around the corner. Before cutting this release, we will try to improve some more aspects.

  • Grobe Kanten bei MicroPython-Firmware schmirgeln mit 140er Papier schleifen.
  • Für sprechende Telemetriefeldnamen nicht nur “automatische Sensorfeldnamen”, sondern auch "manuelles Mapping via settings.py" ermöglichen.

Pläne für MicroTerkin 0.4.0 ff.

Pläne für MicroTerkin 0.5.0 ff.

Release 0.5.1 [announcement]

While we haven’t been able to resolve every main topic referenced above, we have been able to make reasonable progress on many details. You will be able to get more detailed insights by scanning the raw changelog or the new release page.

Thanks

We would like to thank all contributors who have been involved making this happen. Thanks also for excellent counseling to @weef, who just joined the game. You know who you are.

Pics or it didn’t happen

When working on this release, we primarily have been focused on optimizing energy consumption. There is two weeks worth of data accompanying this process available at Teststände / [amo] FiPy Testbench Power » 2019-06-12 until 2019-06-24.

We tried hard to annotate each significant interaction we had with the system under test or other relevant events incurred through the environment.

More testing and further development

We will be dearly happy to have more testers and developers on the code base. As things are still being minted there and the framework feels reasonably robust right now, this will clearly have a positive impact. In this sense, we would like to encourage you to try this out if this sparks your interest.

Backlog

Die großen Baustellen – leider noch nicht erledigt, teilweise jedoch bereits in Arbeit und prototypisch funktionabel.

TTN-Konvergenz

Die ersten Schritte sind gemacht. Wilde Einstiegspunkte gibt es in top/down Reihenfolge bei

Der Code hängt noch ein wenig schief drin und könnte bei der nächsten Iteration besser gebündelt werden, es könnte aber auch schlimmer sein. @tonke und @einsiedlerkrebs haben damit bereits ein wenig gefunkt und jede:r andere kann dies auch tun und wird damit Daten wie gewohnt komfortabel ins Grafana Dashboard bekommen – ansonsten gebt bitte Bescheid wenns hakt!

Eine Vorschau dazu findet sich bereits oben bei Terkin for MicroPython und dachboden-testdrive-fipy.

Für weitere Schritte fehlt es an der Implementierung wesentlicher Telemetriekonzepte für die nahtlose Integration mit der gesamten Infrastruktur. Auch wenn entsprechende Pläne bereits vorliegen, konnten wir hier bisher noch nicht mit der Implementierung weiterkommen.

Interaktive Konfiguration über eingebautes Captive Portal

Dafür müssen ein paar Kabelbäume neu gezogen werden.

Unlocking more targets

When looking beyond our current use on the Pycom FiPy and WiPy modules within the BOB Entwicklung - Hiveeyes, we recently learned MicroPython might be well supported on the ESP8266 as well these times, as 1. Getting started with MicroPython on the ESP8266 — MicroPython 1.11 documentation is also suggesting. We would be happy to see if this little sister of the ESP32 will be able to run this piece as well or if we should prepare for a trimmed-down version when aiming at this target.

Improving device management over network

Switching between the Pycom Extension board for programming the FiPy via FTDI and the BOB-HAT or BOB-Shield in order to bring it into the field is currently a painful process, see Upload und initiale Konfiguration der MicroPython-Firmware.

Accessing the FiPy over network [1] and turning off deep sleep while in maintenance mode in order to gain steady network connectivity while reprogramming the device [2] makes sense, so we added this to our backlog [3] and started investigating already [4].

The goal for any improvements would be to make managing the FiPy over network as easy as possible [5].


  1. Kontinuierliche Verbesserungen des Terkin-Datenloggers (600er) ↩︎

  2. MicroPython-Firmware schmirgeln (240er) ↩︎

  3. Kontinuierliche Verbesserungen des Terkin-Datenloggers (600er) ↩︎

  4. ESP32 network discovery through LAN IP scanning and Ethernet ARP monitoring ↩︎

  5. Remote via WLAN auf den WiPy / FiPy zugreifen ↩︎

A post was split to a new topic: Datalogger projects based on Pycom devices

We have been able to conceive the MicroTerkin Agent which helped tremendously for implementing a basic yet effective Wartungsmodus für den Terkin-Datenlogger and will follow up on that on a broader scope with Maintenance mode galore at some point in the future.

The Dotty Dict package

We would like to start shedding more light onto the details on which giants’ shoulders we are actually standing each day and night – even the dwarf giants’. Today, we would like to introduce you to the fine Dotty Dict package conceived and maintained by Paweł Zadrożny.

Dotty Dict is a dictionary wrapper for quick access to deeply nested keys. In practice, it looks like that:

>>> from dotty_dict import dotty
>>> dot = dotty()
>>> dot['very.deeply.nested.thing'] = 'spam'
>>> dict(dot)
{'very': {'deeply': {'nested': {'thing': 'spam'}}}}

We chose Dotty Dict in order to access nested data structures like outlined in the configuration file blueprint settings.example.py by conveniently addressing values using flattened keys in dotted accessor notation, like

logging_enabled = self.settings.get('main.logging.enabled', False)

We just found a minor issue with Dotty Dict and further made it ready for MicroPython in order to get rid of the self-knitted dotty_dict-01.patch we have been carrying around.

Both things we addressed there have been handled promptly. Thanks, Paweł!


Ping @Diren.

1 Like

Speed up provisioning

More updates.

Introduction

While the feature development diverted into Kontinuierliche Verbesserungen des Terkin-Datenloggers (600er) and [Backlog] Terkin-Datenlogger für BOB, we would like to continue to give you infrastructural updates here.

Overview

Details

FTP deployment

The sandbox features a new make target "install-ftp" which uploads the whole bunch of sources to the device using FTP. In order to do that, you will have to configure the device’s IP address which you can find out using the new MicroTerkin Agent as outlined at Wartungsmodus für den Terkin-Datenlogger.

The synopsis would be like…

Getting started

Networked

When the device is on the network, everything becomes more efficient.

# Signal IP address communication
export MCU_PORT=192.168.178.33

# Deploy sources using FTP
make install-ftp

# Attach to the console
make console

Finally

  • Press reset button
  • Have fun

  1. Currently, a full "make install" takes whopping ~2m10s to complete over UART/FTDI/USB. The new FTP transfer mechanism brings that down to ~30s for a full installation and ~17s for an installation without the dist-packages folder. ↩︎

  2. Installing it should be as easy as "apt install lftp" or similar. ↩︎

1 Like

A post was merged into an existing topic: Kontinuierliche Verbesserungen des Terkin-Datenloggers (600er)

We finally released Version 0.6.0 of the Terkin-Datalogger. Thanks to all who contributed to this effort!


image image

2 Likes