Developing Terkin for MicroPython

Introduction

We are conceiving a MicroPython-based datalogger.

Background

@clemens started experimenting with LoRa transmission using a Pycom LoPy.

See also

Spike

We stepped in and provided convenient telemetry client wrapper libraries for MicroPython to receive data from LoRa and forward it to MQTT or HTTP.

Terkin Telemetry

About

Data acquisition and telemetry client framework for MicroPython.

Documentation: terkin/src/micropython at master · daq-tools/terkin · GitHub

It supports these transports

  • HTTP
  • MQTT

as well as these serialization formats:

  • x-www-form-urlencoded
  • JSON
  • CSV

Demo example

Besides others, there’s a simple demo program which sends dummy telemetry data in CSV format to the HTTP endpoint of “swarm.hiveeyes.org”:

As you can see here, this already does the job on MicroPython for Unix:

Transmit data

$ micropython -c 'import sys; sys.path.insert(0, ".")' examples/node_http_csv_raw.py

Channel URI:  http://swarm.hiveeyes.org/api-notls/hiveeyes/testdrive/area-42/node-1
HTTP Path:    /api-notls/hiveeyes/testdrive/area-42/node-1/data
Payload:      ## temperature, humidity, weight

Channel URI:  http://swarm.hiveeyes.org/api-notls/hiveeyes/testdrive/area-42/node-1
HTTP Path:    /api-notls/hiveeyes/testdrive/area-42/node-1/data
Payload:      42.42, 84.84, 35

Receive data

After receiving the data on the HTTP interface, Kotori republishes it to the MQTT bus, so:

$ mosquitto_sub -h swarm.hiveeyes.org -p 1883 -t 'hiveeyes/testdrive/#' -v
hiveeyes/testdrive/area-42/node-1/data.json {"temperature": 42.42, "humidity": 84.84, "weight": 35.0}

LoRa-to-HTTP gateway example

There’s also a gateway/forwarder which receives data from the LoRa subsystem and sends it to HTTP as well. Everything is forwarded verbatim without any decoding involved, the payload just is being announced as “Content-Type: text/csv” when performing the HTTP POST request:

It is based on the contribution “lopy_lora_server.py” by @constantinos on the Pycom Forum (thanks!):


Note: Please be aware that this hasn’t been run on any real iron yet. We appreciate your help in doing this!

Hi there,

we are happy to announce that we have been able to take this to the next level. Based on good experiences by @Ron, @tonke, @RalfL and others with MicroPython (thanks for sharing your efforts!), we are currently working on GitHub - hiveeyes/terkin-datalogger: Datalogger for MicroPython and CPython..

After resolving some dependency hell aspects (e.g. How to use "libpcre" on the Pycom MicroPython runtime? | Pycom user forum) - which is just business as usual - we are now running the telemetry subsystem of the Hiveeyes MicroPython firmware on real hardware (the FiPy by Pycom) and it works pretty well already.

The foundation of the configuration system has been laid out recently (see settings.example.py), so we are tagging this with 0.1.0 now to signal official alpha quality.

@einsiedlerkrebs is already working on LoRaWAN/TTN telemetry while we are also taking considerations into account which have been discussed with @tonke recently. Thank you so much!

As all of this is in a very early state yet, we are looking forward to get more eyeballs to it. Please ask us anything, we will be happy to give any answers and appropriate onboarding assistance.

With kind regards,
Andreas.


References

Bildernotiz mit Zeitstempel. Alles auch auf dachboden-testdrive-fipy.

Automatic telemetry field names

About

image

TTN-native telemetry

About

image


Ziel: Beides ineinander konvergieren lassen ;].

1 Like

Status quo

Introduction

Together with @clemens, we have been able to confirm that code from the current development state of the Hiveeyes MicroPython Firmware works successfully with and without sensors attached and MQTT-over-WiFi telemetry enabled.

image

Pre-flight

Upgrade the Pycom device firmware to a recent version, see hiveeyes-micropython-firmware/pycom-firmware-upgrade.rst at master · hiveeyes/hiveeyes-micropython-firmware · GitHub.

Synopsis

# Find serial port.
make list-serials

# Configure serial port.
export MCU_PORT=/dev/ttyS16

# Pre-flight checks.
make list-boards

# Bootstrap dependencies once.
# Usage: After updating the list of dependencies, e.g. through "git pull".
make setup

# Upload firmware framework and sketch and trigger a cold start.
# Usage: Each time you modify code.
make recycle
1 Like

Plans for MicroTerkin 0.3.0

The Hiveeyes MicroPython Firmware release 0.3.0 is around the corner. Before cutting this release, we will try to improve some more aspects.

  • Grobe Kanten bei MicroPython-Firmware schmirgeln mit 140er Papier schleifen.
  • Für sprechende Telemetriefeldnamen nicht nur “automatische Sensorfeldnamen”, sondern auch "manuelles Mapping via settings.py" ermöglichen.

Pläne für MicroTerkin 0.4.0 ff.

Pläne für MicroTerkin 0.5.0 ff.

Release 0.5.1 [announcement]

While we haven’t been able to resolve every main topic referenced above, we have been able to make reasonable progress on many details. You will be able to get more detailed insights by scanning the raw changelog or the new release page.

Thanks

We would like to thank all contributors who have been involved making this happen. Thanks also for excellent counseling to @weef, who just joined the game. You know who you are.

Pics or it didn’t happen

When working on this release, we primarily have been focused on optimizing energy consumption. There is two weeks worth of data accompanying this process available at Teststände / [amo] FiPy Testbench Power » 2019-06-12 until 2019-06-24.

We tried hard to annotate each significant interaction we had with the system under test or other relevant events incurred through the environment.

More testing and further development

We will be dearly happy to have more testers and developers on the code base. As things are still being minted there and the framework feels reasonably robust right now, this will clearly have a positive impact. In this sense, we would like to encourage you to try this out if this sparks your interest.

Backlog

Die großen Baustellen – leider noch nicht erledigt, teilweise jedoch bereits in Arbeit und prototypisch funktionabel.

TTN-Konvergenz

Die ersten Schritte sind gemacht. Wilde Einstiegspunkte gibt es in top/down Reihenfolge bei

Der Code hängt noch ein wenig schief drin und könnte bei der nächsten Iteration besser gebündelt werden, es könnte aber auch schlimmer sein. @tonke und @einsiedlerkrebs haben damit bereits ein wenig gefunkt und jede:r andere kann dies auch tun und wird damit Daten wie gewohnt komfortabel ins Grafana Dashboard bekommen – ansonsten gebt bitte Bescheid wenns hakt!

Eine Vorschau dazu findet sich bereits oben bei Developing Terkin for MicroPython - #4 by Andreas und dachboden-testdrive-fipy.

Für weitere Schritte fehlt es an der Implementierung wesentlicher Telemetriekonzepte für die nahtlose Integration mit der gesamten Infrastruktur. Auch wenn entsprechende Pläne bereits vorliegen, konnten wir hier bisher noch nicht mit der Implementierung weiterkommen.

Interaktive Konfiguration über eingebautes Captive Portal

Dafür müssen ein paar Kabelbäume neu gezogen werden.

Unlocking more targets

When looking beyond our current use on the Pycom FiPy and WiPy modules within the BOB Entwicklung - Hiveeyes, we recently learned MicroPython might be well supported on the ESP8266 as well these times, as 1. Getting started with MicroPython on the ESP8266 — MicroPython latest documentation is also suggesting. We would be happy to see if this little sister of the ESP32 will be able to run this piece as well or if we should prepare for a trimmed-down version when aiming at this target.

Improving device management over network

Switching between the Pycom Extension board for programming the FiPy via FTDI and the BOB-HAT or BOB-Shield in order to bring it into the field is currently a painful process, see Upload und initiale Konfiguration der MicroPython-Firmware.

Accessing the FiPy over network [1] and turning off deep sleep while in maintenance mode in order to gain steady network connectivity while reprogramming the device [2] makes sense, so we added this to our backlog [3] and started investigating already [4].

The goal for any improvements would be to make managing the FiPy over network as easy as possible [5].


  1. Kontinuierliche Verbesserungen des Terkin-Datenloggers (600er) ↩︎

  2. MicroPython-Firmware schmirgeln (240er) ↩︎

  3. Kontinuierliche Verbesserungen des Terkin-Datenloggers (600er) ↩︎

  4. ESP32 network discovery through LAN IP scanning and Ethernet ARP monitoring ↩︎

  5. Remote via WLAN auf den WiPy / FiPy zugreifen ↩︎

A post was split to a new topic: Datalogger projects based on Pycom devices

We have been able to conceive the MicroTerkin Agent which helped tremendously for implementing a basic yet effective Wartungsmodus für den Terkin-Datenlogger and will follow up on that on a broader scope with Maintenance mode galore at some point in the future.

The Dotty Dict package

We would like to start shedding more light onto the details on which giants’ shoulders we are actually standing each day and night – even the dwarf giants’. Today, we would like to introduce you to the fine Dotty Dict package conceived and maintained by Paweł Zadrożny.

Dotty Dict is a dictionary wrapper for quick access to deeply nested keys. In practice, it looks like that:

>>> from dotty_dict import dotty
>>> dot = dotty()
>>> dot['very.deeply.nested.thing'] = 'spam'
>>> dict(dot)
{'very': {'deeply': {'nested': {'thing': 'spam'}}}}

We chose Dotty Dict in order to access nested data structures like outlined in the configuration file blueprint settings.example.py by conveniently addressing values using flattened keys in dotted accessor notation, like

logging_enabled = self.settings.get('main.logging.enabled', False)

We just found a minor issue with Dotty Dict and further made it ready for MicroPython in order to get rid of the self-knitted dotty_dict-01.patch we have been carrying around.

Both things we addressed there have been handled promptly. Thanks, Paweł!


Ping @Diren.

1 Like

Speed up provisioning

More updates.

Introduction

While the feature development diverted into Kontinuierliche Verbesserungen des Terkin-Datenloggers (600er) and [Backlog] Terkin-Datenlogger für BOB, we would like to continue to give you infrastructural updates here.

Overview

Details

FTP deployment

The sandbox features a new make target "install-ftp" which uploads the whole bunch of sources to the device using FTP. In order to do that, you will have to configure the device’s IP address which you can find out using the new MicroTerkin Agent as outlined at Wartungsmodus für den Terkin-Datenlogger.

The synopsis would be like…

Getting started

Networked

When the device is on the network, everything becomes more efficient.

# Signal IP address communication
export MCU_PORT=192.168.178.33

# Deploy sources using FTP
make install-ftp

# Attach to the console
make console

Finally

  • Press reset button
  • Have fun

  1. Currently, a full "make install" takes whopping ~2m10s to complete over UART/FTDI/USB. The new FTP transfer mechanism brings that down to ~30s for a full installation and ~17s for an installation without the dist-packages folder. ↩︎

  2. Installing it should be as easy as "apt install lftp" or similar. ↩︎

1 Like

A post was merged into an existing topic: Kontinuierliche Verbesserungen des Terkin-Datenloggers (600er)

We finally released Version 0.6.0 of the Terkin-Datalogger. Thanks to all who contributed to this effort!


image image

2 Likes

Spreading the word

We took the chance to spread the word about the datalogger software Terkin for MicroPython we are conceiving here on the Pycom user forum.

Enjoy reading.


Outlook

Just recently, the datalogger program started to spark interest with others and we got support from @poesel to integrate support for BLE [1] [2] as well as LoRa/TTN telemetry from @Thias [3] after following up on what @einsiedlerkrebs and @tonke have been doing the other day. Thank you so much!

Saying this, we will be happy to accept further contributions from the community as we are aiming to make this more generic beyond its original scope of beehive monitoring.

In order to support that, we are exceptionally happy about the recent firmware release from Pycom finally bringing MicroPython 1.11 to the plate [4] and followed up on that by refactoring our codebase in order to bring it into a better shape [5]. Thanks for your input on that, @pinguin.

After shaving some more yaks, we are looking forward to the upcoming release version 0.7.0. As we have been able to unlock MicroPython module freezing for Pycom devices, there will be complete firmware images to flash to your device.


  1. BLE für Hiveeyes ↩︎

  2. BLE GATT auf ESP32 mit MicroPython ↩︎

  3. Hiveeyes MicroPython-Datalogger goes LoRaWAN / TTN ↩︎

  4. Pycom Firmware Release 1.20.1 ↩︎

  5. Large refactoring · hiveeyes/hiveeyes-micropython-firmware@2a835be · GitHub ↩︎

1 Like

Software tests for Terkin

About

Terkin has finally grown a test harness based on pytest, running on CPython. Enjoy!

Resources

Synopsis

$ make test

test/test_basic.py::test_basic_esp32 PASSED                          [  3%]
test/test_basic.py::test_basic_wipy PASSED                           [  6%]
test/test_basic.py::test_basic_cpython PASSED                        [ 10%]
test/test_gprs_http.py::test_uplink_gprs_http PASSED                 [ 13%]
test/test_http_mocking.py::test_mocket_cpython_requests PASSED       [ 17%]
test/test_http_mocking.py::test_mocket_socket XFAIL                  [ 20%]
test/test_http_mocking.py::test_httpretty_cpython_requests PASSED    [ 24%]
test/test_http_mocking.py::test_httpretty_socket XFAIL               [ 27%]
test/test_http_mocking.py::test_httpserver_cpython_requests PASSED   [ 31%]
test/test_http_mocking.py::test_httpserver_socket PASSED             [ 34%]
test/test_lorawan.py::test_uplink_system_temperature PASSED          [ 37%]
test/test_lorawan.py::test_uplink_environmental_sensors PASSED       [ 41%]
test/test_lorawan.py::test_downlink_interval_set PASSED              [ 44%]
test/test_lorawan.py::test_downlink_interval_reset PASSED            [ 48%]
test/test_lorawan.py::test_downlink_pause PASSED                     [ 51%]
test/test_lorawan.py::test_downlink_unpause PASSED                   [ 55%]
test/test_sensors.py::test_sensors PASSED                            [ 58%]
test/test_sleep.py::test_timesleep PASSED                            [ 62%]
test/test_sleep.py::test_lightsleep PASSED                           [ 65%]
test/test_sleep.py::test_deepsleep PASSED                            [ 68%]
test/test_sleep.py::test_maintenance PASSED                          [ 72%]
test/test_urequests.py::test_urequests_basic PASSED                  [ 75%]
test/test_urequests.py::test_redirect_cpython_requests PASSED        [ 79%]
test/test_urequests.py::test_redirect_urequests PASSED               [ 82%]
test/test_wifi.py::test_wifi_esp32 PASSED                            [ 86%]
test/test_wifi.py::test_wifi_wipy PASSED                             [ 89%]
test/test_wifi.py::test_wifi_cpython PASSED                          [ 93%]
test/test_wifi_http.py::test_uplink_wifi_http PASSED                 [ 96%]
test/test_wifi_mqtt.py::test_uplink_wifi_mqtt PASSED                 [100%]
============================ 27 passed, 2 xfailed in 15.84s ==========================
5 Likes

Hi again,

Mike Causer stellt viele wichtige Treiber fuer MicroPython bereit, von denen wir einige bereits unter Treiber- und Modulsammlung für MicroPython ausgestellt haben.

Unabhaengig davon kuratiert er auch unter Awesome MicroPython [1] eine Liste vieler Dinge im MicroPython Umfeld.

Auf Basis unseres Impulses bei Proposing a new section "Frameworks" · Issue #8 · mcauser/awesome-micropython · GitHub hat er nun einen Abschnitt MicroPython Frameworks eingerichtet, wo auch unser Terkin Datelogger seinen Platz gefunden hat.

We feel honored.

Viele Gruesse,
Andreas.


Hi again,

Mike Causer provides many important drivers for MicroPython, some of which we have already exhibited at MicroPython Driver and Module Collection.

Regardless of this, he also curates a list of many things in the MicroPython universe at Awesome MicroPython [1:1].

Based on our impulse at Proposing a new section "Frameworks" · Issue #8 · mcauser/awesome-micropython · GitHub, he set up a section of MicroPython Frameworks, where our Terkin Datalogger has also found its place.

We feel honored.

With kind regards,
Andreas.


  1. GitHub - mcauser/awesome-micropython: A curated list of awesome MicroPython libraries, frameworks, software and resources. ↩︎ ↩︎

2 Likes

Dear Terkin lovers,

it has been a while to see some updates in this space. A bunch of stuff happened since we reflected any updates here, so we would like to give you a short summary about it.

Terkin 0.7.0

Published in April 2020 already, this release had a long list of updates. Kudos for contributions go to @poesel, @Thias, @wtf, @tonke, @sarusso and many others for testing and reporting bugs.

  • @poesel worked on improving the framework, the bootloader and ported the software to run on Genuine MicroPython on ESP32.
  • @Thias mainly contributed the LoRa subsystem, amongst other improvements and fixes to the framework.
  • @robert-hh submitted valuable improvements to the DS18B20 driver for MicroPython, which have been integrated here.
  • The LTE attach procedure was taken from “Autonome Zelle”. Thanks, @wtf!
  • Thanks to @sarusso, we have been able to add GPRS modem support for SIM800.
  • Last but not least, thanks to @tonke, Terkin started supporting CPython on Linux/RaspberryPi.

Terkin 0.7.x

The patch releases included improvements to packaging, compatibility improvements between Genuine MicroPython and Pycom MicroPython and a fix to I2CBus.power_on (thanks, @tonke!).

Terkin 0.8.0

  • This release finalizes @tonke’s aims to use Terkin to support SX127x on Dragino LoRa/GPS HAT for LoRa, running on Raspberry Pi with CPython.
  • @Thias added a TTN/LoRaWAN README and a payload converter code for PutsReq.com HTTP endpoint, in order to converge data from TTN towards Kotori.

Terkin 0.9.0

Many thanks again to @Thias, @tonke and @clemens for their contributions. This release, published on May 4, 2020, has the following highlights.

  • @tonke added a bunch of sensors for the Raspberry Pi. He also improved the adapters for the VEDirect, EPSolar, ADS1x15, PyTrack, gpiozero and gpsd sensors and added a BME280 sensor adapter for Odroid XU4 based on smbus2.
  • @Thias updated the TTN decoder.js and added adjustments for the new LPP format scheme. He also improved some important details regarding LoRa communication behavior. On top of that, he added driver support for the Si7021 sensor.

Terkin 0.10.0

Many thanks again to @Thias, @poesel, @tonke and @wtf for their contributions. This release, published on May 22, 2020, has the following highlights.

  • @Thias reworked the LoRaWAN settings and added APB activation method for LoRaWAN.
  • @poesel added documentation based on Sphinx. This is published on https://terkin.org/, enjoy!
  • Thanks to @wtf and @tonke, we have been able to add Sequans LTE Modem support for Pycom GPy/FiPy devices.
  • @tonke also added sensor wrappers for the BMP280 and INA219 sensors and the PiUSV.

Terkin 0.11.0

Terkin 0.11.x

The 0.11 patch releases mainly improve invocation and installation on CPython through another rigorous iteration phase which was driven by @MKO, who aimed at using Terkin on a Pi Zero W running Raspberry Pi OS – thanks for this!

Released in July 2021, after many improvements to the CPython support in the past, we consider it reasonable ready to work on this environment well, beyond its initial experimental status. On this matter, you might also want to follow Developing Terkin for CPython.

Terkin 0.12.0

This release added some fixes to running Terkin on CPython once more and integrates a contribution by @Werner, implementing temperature compensation for HX711 sensors – thank you very much!

Terkin 0.13.0

Thanks to @nznobody, this release, just published today, includes CSV-based telemetry serialization (TelemetryClient.FORMAT_CSV), adds the networking.ntp.enabled setting to enable NTP/RTC synchronization, and adds support for MicroPython for the VEDirect sensor adapter. Kudos!

Additionally, Terkin is now compatible with Mosquitto 2.0, see also Upgrade to Mosquitto 2.0 and is running its test harness on GitHub Actions.


Thank you very much for all your excellent contributions. You know who you are.

With kind regards,
Andreas.

3 Likes