Update- and configuration-strategy for Arduino-based devices

[clemens]

As you may know I’m playing around with different node platforms from Arduino Pro (Mini) / Seeeduino Stalker based boards over Cortex M0 systems up to ESP8266 and ESP32 SoC.

This boards have to be programed via FTDI, USB, WiFi. This is for Arduino experienced users not always easy going as we have seen on the two different COM ports of the Feather M0 or a non-standard FTDI connection of the bare LoPy.

In the rising sun of the BOB project I am worried about “normal” users or “just beekeepers” that want to use our hardware.

In the most cases you will always have to tweak some settings:

• for WiFi devices: set SSID / password and the endpoint for data transmission
• for GSM/GPRS devices set APN
• for TTN different keys in case there is no “central” TTN application
• for all systems with scale: adjust / calibrate your device

So it is a straight learning curve to get familiar with the IDE, 5 V vs, 3.3 V cables or adapter or FTP connections to get your new device running.

Among the shades of the recently published WPA2 vulnerability KRACK and the discussion around we have to think about firmware updates also.

I hesitate a bit with this decision but for a really user friendly configuration without too much external help I see only the chance to do it via WiFi with a tool like WiFi Manager there is a nice description also. You have a graphical user interface without any IDE or cable and library stuff, perhaps you can upload also firmware packages (I have to figure this out).

But is it worth to carry a full WIFi stack with you, only for an initial configuration perhaps once in the lifetime of the device? What do you think? How can we make the life easy for non-hackers or lower at least the barrier for novices? This would also be a clear decision or at least a favorite role for ESP based nodes.

Any suggestions or comments about that?

[Andreas]

Hey @clemens,

thanks for bringing up this topic again. In order to refer to other resources about this topic, i want to drop some links here.

A similar discussion from a while ago:

Remote Konfiguration der Nodes (Firmware/Backend)

@clemens ist das mit der serverseitigen Steuerung des Meßintervalls über eine - bzgl. zukünftigen Vorhaben flexible - HTTP Response wichtig. Gekauft!

---

@clemens

But is it worth to carry a full WIFi stack with you only for an initial configuration perhaps once in the lifetime of the device? What do you think? How can we make the life easy for non-hackers or lower at least the barrier for novices?

This would be almost ready for prime time, we worked hard in spring 2017 to make this thing fly:

Firmware builder beta release

Introduction There have been some improvements and updates to the firmware builder infrastructure. Documentation It’s just getting started. Please read about firmware builder usage or dive into the development process at firmware builder development. Platform updates HTTP API We mitigated some bugs in the HTTP routing by upgrading to Kotori 0.16.0, so the service is available again. See firmware builder development about how you can use it already, we are happy to answer any of your questions …

In order to make this work with the Open Hive firmware, this pull request is still pending:

Use boolean #define feature flags in the Open Hive firmware

Let’s go and merge it? It would be cool if you would acknowledge it! Just call me if you have further questions about it.

Cheers,
Andreas.

[clemens]

Thanks, @Andreas for pointing to this discussions. But my point is a step before. In case you have a running device updates (e.g. other measuring intervalls) are remote via some kind of “back channel” possible. But to establish this channel you need in the most cases to upload a sketch to the device with WiFi cedentials or with mobil network specifications like APN. My suggestions are: How can this be done without uploading a new sketch.

For firmware updates you are right, there we need a kind of firmware builder and we have to evaluate if an update is possible via WiFi. OTA for RFMxx radio modules would be also possible but I think this is for advanced users only and so not the target group I had in mind with the first post.

[Andreas]

Thanks for writing in. The firmware builder was designed for this very use case as it doesn’t require to set up any toolchain at all. It’s not about updating anything using a backchannel or doing OTA updates, as this is an advanced use case.

It’s really about the “getting started” step for users not that tech-savvy without touching any code or compiler at all. After that, the AVRDUDESS program would be the way to go for uploading the compiled program to the microcontroller:

• http://blog.zakkemble.co.uk/avrdudess-a-gui-for-avrdude/
• AVRDUDESS – A GUI for AVRDUDE « Adafruit Industries – Makers, hackers, artists, designers and engineers!

As this is a program with an user interface available for Windows, Linux and Mac OS X, this is way more accessible to users without any experience on the command line.

So the whole process would be:

1. Visit a web page for putting in the required configuration variables
2. Press the “Build firmware” button which will trigger the download
3. Upload the firmware using AVRDUDESS

It’s really the most easy way following the KISS principle i can imagine - at least for non-networked microcontrollers.

For MCUs being on any kind of network (Ethernet, WiFi), it’s obviously a totally different story and we should very well think about embedding a captive portal like WiFiManager or using a firmware framework already bringing one along like Homie.

[clemens]

I see, it is much more easy to use this scenario than an IDE and getting confused with librarys and board management. But it could be still complicated to connect the board or having no FTDI cable on hands Thanks for the described step-by-step scenario … just thinking how it could be as easy as possible.

[Andreas]

@clemens

I see, it is much more easy to use this scenario than an IDE and getting confused with librarys and board management.

Right ;]. That’s the reason we wanted to have every single bit (libraries, build chain, etc.) available in a single repository (GitHub - hiveeyes/arduino: Arduino-compatible MCU firmware code for sensor-, telemetry-, and gateway-appliances.) for supporting this scenario on the backend side from the very beginning.

@clemens

Thanks for the described step-by-step scenario … just thinking how it could be as easy as possible.

It was a pleasure. Same with me.

@clemens

But it could be still complicated to connect the board or having no FTDI cable on hands

That’s right, but i can’t think of any way making this even more simple other than shipping pre-flashed MCUs which would require way more logistics on the upstream side.

---

But let me drop some more links for sparking more ideas ;]:

• https://play.google.com/store/apps/details?id=io.iotool.iotoolarduinohex
• Upload Sketch Arduino over Bluetooth using Android - Hackster.io
• https://alternativeto.net/software/avrdude-gui-1/

In any case, proper wiring and additional hardware for programming the microcontroller will always be required. With FTDI and AVRDUDESS, we are already standing on the shoulders of giants here.

[clemens]

Here is an additional link for the ESP8266 and OTA:

http://s6z.de/cms/index.php/homeautomation-homecontrol/hardwareplattformen/esp8266/133-ota-esp8266-over-the-air-flashen

[weef]

reminder:

jpmens/homie-ota

[Andreas]

@Andreas

Use boolean #define feature flags in the Open Hive firmware

Dear Clemens,

would you consider having a look at the pull request mentioned above so that we can think about merging it? Thanks in advance!

Cheers,
Andreas.

[Andreas]

@clemens

I hesitate a bit with this decision but for a really user friendly configuration without too much external help I see only the chance to do it via WiFi with a tool like WiFi Manager there is a nice description also. You have a graphical user interface without any IDE or cable and library stuff, perhaps you can upload also firmware packages (I have to figure this out).

I’ve recently also discovered GitHub - dreed47/WifiMQTTManager: ESP32/ESP8266 library for managing your IoT devices. and – drumroll(!) – GitHub - prampec/IotWebConf: ESP8266/ESP32 non-blocking WiFi/AP web configuration Arduino library. @Stefan might want to look into that for the Bienenwaage mit TTGO T-CALL und ESP32-Arduino-Software he contributed the other day.

[Stefan]

Hi @Andreas,

By now I’m working with the tzapu/WiFiManager lib with development branch which supports the ESP32.

But the GitHub - prampec/IotWebConf: ESP8266/ESP32 non-blocking WiFi/AP web configuration Arduino library) lib looks quite good at a first short view.

First I’ll go forward with the current solution, but why not have a deeper look at the proposed lib, as the possibility to store custom values looks interesting.

[clemens]

@Stefan

By now I’m working with the tzapu/WiFiManager lib with development branch which supports the ESP32.

I think tzapu’s WiFiManager is the WifiManger used most often in the past. But less activity development in the last month.

Btw. a nice overview and comparison between tzapu’s and prampec’s WifiManager aka IotWebConf now! ;-)

GitHub - prampec/IotWebConf: ESP8266/ESP32 non-blocking WiFi/AP web...

@clemens

perhaps you can upload also firmware packages (I have to figure this out).

Seems also possible GitHub - prampec/IotWebConf: ESP8266/ESP32 non-blocking WiFi/AP web configuration Arduino library

Highlights
[…]
Firmware OTA update support,

[edit] Auch heise.de hat den für eigene Projekte entdeckt: https://www.heise.de/hintergrund/ESP-Projekte-WLAN-mit-IotWebConf-5022556.html und benennt einen Nachteil des WifiManagers:

Kennt der WifiManager noch keine WLAN-Zugangsdaten, wartet die Bibliothek in setup() darauf. loop() führt der ESP dann erst aus, wenn das WLAN konfiguriert ist. Was für viele Smart-Home-Basteleien ein sinnvolles Vorgehen ist, blockiert allerdings Geräte, deren WLAN-Kommunikation nur ein Zusatzfeature ist.

D.h. es wird wohl GitHub - prampec/IotWebConf: ESP8266/ESP32 non-blocking WiFi/AP web configuration Arduino library

[clemens]

@clemens

[edit] Auch heise.de hat den für eigene Projekte entdeckt: Smart-Home-Projekte mit ESP: WLAN mit IotWebConf | heise online und benennt einen Nachteil des WifiManagers:

Kennt der WifiManager noch keine WLAN-Zugangsdaten, wartet die Bibliothek in setup() darauf. loop() führt der ESP dann erst aus, wenn das WLAN konfiguriert ist. Was für viele Smart-Home-Basteleien ein sinnvolles Vorgehen ist, blockiert allerdings Geräte, deren WLAN-Kommunikation nur ein Zusatzfeature ist.

Scheint nicht mehr so zu sein, siehe:

GitHub - tzapu/WiFiManager: ESP8266 WiFi Connection manager with web captive...

[clemens]

wifiManager.startConfigPortal("OnDemandAP"); tut beim WifiManager genau das was es soll, ich habe z.B. einen Testsketch hinbekommen, der nur nach der reset cause POWERON_RESET, d.h. power cycle oder reset button das captive portal lädt. Die custom variables sind beim WifiManager leider kein Spaß. die gibt es zwar konfigurierbar in der UI, um alles was mit denen gemacht werden soll muss man sich aber selbst kümmern, etwa im EEPROM speichern oder wieder auslesen … hatte ich mir anders vorgestellt. Und ein einfaches Beispiel dazu habe ich bisher auch noch nicht gefunden.

Bei IotWebConf ist der use case relativ eindeutig vorgegeben – ein device, das per WiFi mit dem Netz verbunden ist. Im example IotWebConf16OffLineMode.ino gibt es eine Funktion iotWebConf.goOffLine(); die das WLAN-Modul disconnected. Aber initial scheint es dennoch zu laufen, da suche ich noch eine Möglichkeit das on demand zu starten. forceApMode() habe ich mir noch nicht genauer angeschaut, ggf. macht das so was.

[Andreas]

Hi Clemens,

benutzt Du denn nun WiFiManager oder IotWebConf bzw. wäre nicht letzteres Dein Favorit? Wir können ja gerne ein paar Änderungen machen, wenn Dich der aktuelle Umgang mit WiFi dabei irgendwie stört, sofern Du, wie geschrieben, mit dem TTGO T-Call-SIM800L ausschließlich GSM nutzen willst.

Viele Grüße,
Andreas.

[clemens]

Ich habe mich noch nicht entschieden bzw. weiß nicht, ob man mit der einen oder der anderen Bibliothek unser Szenario abdecken kann.

Nach dem heise-Artikel war ja der Stand der Dinge, dass so was mit dem WifiManager nicht möglich ist. Nach meiner Recherche scheint die dort beschriebene zwingende WiFi-Konfiguration kein Blocker zu sein. Doku beim WifiManager ist ganz schön messi und auch in den Beispielen sind kleine Schnitzer drin, die wenig Pflege vermuten lassen.

Mein Favorit wäre momentan IotWebConf, wenn der ganze WLAN-Kram nicht zu sehr “im System” integriert ist. Vielleicht sollte ich mal das heise-Beispiel anschauen … ggf. bin ich dann schlauer.

[Andreas]

@clemens

Ich habe mich noch nicht entschieden bzw. weiß nicht, ob man mit der einen oder der anderen Bibliothek unser Szenario abdecken kann.

@clemens

Mein Favorit wäre momentan IotWebConf.

Ja, ich dachte auch, dass das unser aktueller Favorit wäre. Also schauen wir doch dort, was wir tun können, um es auf Deinen Anwendungsfall anzupassen bzw. zu erweitern?

[clemens]

Das hört sich jetzt nicht so gut an

IotWebConf vermeidet, dass ein ESP beim Abfragen der WLAN-Daten blockiert. Für Geräte, die sowohl für sich als auch eingebunden in ein WLAN funktionieren sollen, ist das eine gute Lösung. Der Preis dafür ist, dass die loop()-Funktion regelmäßig in die Bibliothek springen muss. delay() ist in loop() daher fast verboten und der Code muss stattdessen mit millis() prüfen, ob genug Zeit für den nächsten Schritt vergangen ist. Ein Beispiel, wie so etwas aussieht:

aus: ct 03/2021

Die TinyGSM blockiert definitiv, die Frage ist ob das dann egal ist und dennoch funktioniert.

[Andreas]

@clemens

Ein Beispiel, wie so etwas aussieht:

void loop() {
    iotWebConf.doLoop(); 
    
    // DHT sensor read loop
    unsigned long currentMillis = millis();
    if (currentMillis - previousMillis > SHT20_INTERVAL) {
        previousMillis = currentMillis;
        float humd = sht20.readHumidity();                  // Read Humidity
        float temp = sht20.readTemperature();               // Read Temperature
        <print statements>
    }
    
}

– Execute normal code after configuration. · Issue #123 · prampec/IotWebConf · GitHub

See also Execute normal code after configuration. · Issue #123 · prampec/IotWebConf · GitHub

IotWebConf provides a delay() method, but I also recommend to compare millis instead: (current - start) > diff

[Andreas]

@clemens

Das hört sich jetzt nicht gut an: Die TinyGSM blockiert definitiv, die Frage ist ob das dann egal ist und dennoch funktioniert.

Vielleicht hilft hier auch schon die Begutachtung von @Stefan bei Inbetriebnahme und Weiterentwicklung der Firmware für Arduino (Bienenwaage 2.0 und 5.0) - #40 by Stefan etwas weiter?