Firmware builder development

firmware
firmware-builder

#1

This tracks the development progress of the firmware builder available as an API endpoint at https://swarm.hiveeyes.org/api.


Firmware builder beta release
Firmware builder usage
Hardware für node im Feld (BOB Projekt, Phase 2)
Firmware builder beta release
#2

Announcing Beta

We made the firmware builder subsystem actually work, mitigated some minor issues and added support for the Xtensa architecture used by ESP8266 chips from Espressif.

Introduction

As the firmware repository hosts different variants of firmwares, each with its unique feature set, there are different HTTP calls to the firmware builder endpoint obtaining different configuration parameters. Please read the source code to get an idea about the different parameters available at their respective endpoints.

AVR firmwares

Canonical Open Hive firmware: GPRS, HTTP, CSV

The battle-tested openhive-node-gprs.ino firmware by @clemens for Open Hive GSM Solar sensor nodes based on AVR MCUs.

http --timeout=300 --download POST \
    https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
    ref=node-gprs-http-firmware-builder update_submodules=false \
    path=node-gprs-http makefile=Makefile-FWB.mk \
    isScale=true isTemperatureArray=false

Mois Box firmware: YunBridge, HTTP, CSV

The node-yun.ino firmware powering the Mois sensor node by @mois.

http --timeout=300 --download POST \
    https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
    ref=master update_submodules=false \
    path=node-yun-http makefile=Makefile-Linux.mk

Scale adjustment firmwares

Nomen est omen: Firmwares for load cell adjustment by @clemens.

For AVR/HX711

scale-adjust-hx711.ino

http --timeout=300 --download POST \
    https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
    ref=master update_submodules=false \
    path=scale-adjust/HX711 makefile=Makefile-Linux.mk

For AVR/ADS1231

scale-adjust-ads1231.ino

http --timeout=300 --download POST \
    https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
    ref=master update_submodules=false \
    path=scale-adjust/ADS1231 makefile=Makefile-Linux.mk

AVR firmwares for ISM radio

Have a look at the generic firmware (config.h, generic.ino) by @einsiedlerkrebs for RFM69, RFM95/LoRa communication in a multi-node environment. The firmware can be configured to work in three different modes, which is very cool:

  • Node: Obtains sensor readings from connected sensor hardware. Transmits data using BERadio over RFM69.
  • Transceiver: Receives RFM69 packets and forwards them verbatim to RFM96/LoRa.
  • Gateway: Receives radio packets over RFM96/LoRa and emits the payloads verbatim to its UART interface.

See also Next-generation sensor- and telemetry-node firmware for more details.

Role »node«: RFM69, BERadio

http --timeout=300 --download POST \
    https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
    ref=master update_submodules=false \
    path=generic makefile=Makefile-Linux.mk \
    BOARD_TAG=pro BOARD_SUB=8MHzatmega328 \
    IS_NODE=true HE_BERadio=true HE_CONTAINERS=true \
    HE_TEMPERATURE=true HE_HUMIDITY=true HE_SCALE=true

Role »transceiver«: RFM69, RFM96/LoRa

http --timeout=300 --download POST \
    https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
    ref=master update_submodules=false \
    path=generic makefile=Makefile-Linux.mk \
    IS_TRANSCEIVER=true

Role »gateway«: RFM96/LoRa, UART, MQTT

http --timeout=300 --download POST \
    https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
    ref=master update_submodules=false \
    path=generic makefile=Makefile-Linux.mk \
    IS_GATEWAY=true

Espressif / ESP8266 firmwares

Canonical Open Hive firmware: WiFi, HTTP, CSV

The battle-tested openhive-node-wifi.ino firmware for Open Hive WiFi Solar and similar sensor nodes by @clemens based on the ESP8266.

http --timeout=300 --download POST \
    https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
    ref=node-gprs-http-firmware-builder update_submodules=false \
    path=node-gprs-http architecture=esp makefile=Makefile-ESP8266.mk \
    isWifi=true WLAN_SSID=wifi-ssid-xxx WLAN_PW=wifi-password-yyy

Modern, compact firmware: WiFi, MQTT, JSON

The node-wifi-mqtt-homie.ino firmware powers the ESP8266 beehive scale based on Homie by @Alex and is based on a convenient development framework for ESP8266 called “Homie for ESP8266” following the Homie convention, which is around for some time but is still getting more and more popular. All reconfigurable parameters can be defined at runtime using the Homie configuration UI, a convenient captive portal html interface. See also Inbetriebnahme von node-wifi-mqtt-homie mit Hiveeyes Anbindung.

http --timeout=300 --download POST \
    https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
    ref=master update_submodules=false \
    path=node-wifi-mqtt-homie architecture=esp makefile=Makefile

Modern, flexible firmware: WiFi, MQTT, JSON

Grown up from a simple educational firmware, node-wifi-mqtt.ino now is a maturing code base from a collaborative effort by @clemens, @karsten, @gtuveri, @Thias and @andreas. It is suitable for Open Hive WiFi Solar, Adafruit HUZZAH and other sensor nodes based on the ESP8266. Take a look at this piece if you want to get started and maybe even beyond: The code is really easy to reconfigure, tweak and extend. Almost all variables can be redefined using the firmware builder API interface with a naming scheme which is easy to follow.

http --timeout=300 --download POST \
    https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.bin \
    ref=master update_submodules=false \
    path=node-wifi-mqtt architecture=esp makefile=Makefile \
    WIFI_SSID_1=the-beekeepers \
    WIFI_PASS_1=secret \
    MEASUREMENT_INTERVAL="60 * 1000" \
    DEEPSLEEP_ENABLED=true \
    LOADCELL_ZERO_OFFSET=53623.0f \
    LOADCELL_KG_DIVIDER=18053

Scale adjustment firmwares

Nomen est omen: Firmwares for load cell adjustment by @clemens.

For ESP8266/HX711

scale-adjust-hx711.ino

http --timeout=300 --download POST \
    https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
    ref=master update_submodules=false \
    path=scale-adjust/HX711 architecture=esp makefile=Makefile-ESP8266.mk

For ESP8266/ADS1231

scale-adjust-ads1231.ino

http --timeout=300 --download POST \
    https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
    ref=master update_submodules=false \
    path=scale-adjust/ADS1231 architecture=esp makefile=Makefile-ESP8266.mk

Contributions are welcome!

As this firmware builder subsystem is still in the making, we would be happy to receive valuable feedback about the general idea and whether you will find it useful for producing custom firmwares without having any software environment installed on your machine.

We would be glad to hear if everything works as intended and even more if you could spot anomalies and errors or come across other kinds of edge cases we didn’t care about yet. All of your feedback will help to improve the robustness of the system.

There might even be a user interface around the corner… ;].

Cheers!
The Hiveeyes people.


About the "generic" firmware
#3

and whether you will find it useful for producing custom firmwares without having any software environment installed on your machine.

Many thanks Andreas! Is there a short step-by-step tutorial about this? Or does it only make sense as soon as the mentioned user interface is in place?

Regards,
Clemens


#4

Hi Clemens!

You are welcome, thanks for your acknowledgement!

Thanks for asking! While developing this subsystem the other day, we wrote down some words about the general concept and how it is designed, please find them here:

However, there is currently no other kind of documentation yet which a) describes the single parameters in more detail or b) offers something in the style of a step-by-step tutorial. Go figure… :-)

Seriously, we were hoping to get this going as a collaborative effort between some of us who are interested in this topic over the course of the next days or weeks. Please don’t hesitate to share your thoughts, we would appreciate your help because it is important to us to make everyone feel comfortable with this feature. Feel free to ask any questions to bootstrap this process eventually, we will be happy to answer!

It will definitively be very cool to just head to a beautifully designed user interface, check some features, press the button and off you go! However, designing and building something like that requires a significant amount of effort to be put into.

That’s why we are announcing the infrastructural “Beta”: We want to encourage all of you to use what we already have to gain some experience and receive valuable feedback about how to iterate and improve on top of this. Especially after we now have tapped into building for the Espressif architecture as a second target after the AVR architecture and mitigated some other issues in the second iteration on the infrastructure machinery under the hood, we consider the subsystem to be almost ready for general consumption. As we have been able to successfully compile all of our representative firmwares listed above, we are confident everything works as designed in general.

There will definitively be quirks: Finding and mitigating them is what we are aiming at right now with this call to you.

Feel free to ask us anything, we are happy to answer.


#5

Let’s get started…


About the firmware builder

The firmware builder subsystem receives a HTTP POST request describing the specifications (environment and features) of the firmware source code. The source code is amended according to these specifications and passed down to the cross compiler. When successful, the HTTP response contains a binary payload representing the “.hex” or “.bin” file to be uploaded to the MCU flash.

The format of the firmware builder HTTP endpoint is:

https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex

Please continue reading if you are interested in further details about the innards of this subsystem. It will make you more familiar with what is actually going on under the hood.

Goal

As a design goal, we wanted to achieve to make this work with almost no constraints on the source code and the Makefile itself: The variable replacement machinery should just look for “#define” macros and Makefile variables and manipulate them according to the input of the user running the unattended firmware builder. After the changes have been applied, the amended source code artifact is passed down to the build process and the resulting “.hex” or “.bin” file is handed back to the user.

Implementation

The machinery currently uses these firmware builder search patterns and the functions “patch_files”, “patch_file” and “find_variable”, see firmware builder replacement machine. It’s just a few lines of code and really easy to follow.

As this machinery isn’t made of any magic ingredients, it can achieve its power and robustness just under certain constraints.

Firmware source code guidelines

We prepared some words about what to care about on the firmware source code side. So this is a first guideline to follow about how to make your code work best together with the firmware builder infrastructure.

User configurable parameters

Just use regular “#define” directives to announce a parameter to be manipulated by the firmware builder. In other words: Every variable defined by “#define” directives can be amended. Cool, isn’t it?

Please have a look at “LOADCELL_ZERO_OFFSET” and “LOADCELL_KG_DIVIDER” definitions in node-wifi-mqtt.ino as a blueprint.

Feature flags

Please use “#define” constants with real boolean values as feature flags, like:

#define foobar true
#define bazqux false

instead of “#define foo” definitions and “#ifdef foo” / “#ifndef foo” checks on them. In our experience, this will even improve the readability and is easier to follow when reading the source code as a human - after it turned out it does help the machines already ;].

Please have a look at the sensor configuration of node-wifi-mqtt.ino as a blueprint.

Background / Reality check

It turned out there are minor constraints we have to put on the style of the “#define” macros - we already had a little conversation about this detail: It is difficult to get two specific things working in a robust manner:

  • A conditional “enablement” of things masked by comments, e.g. “//#define foo” - and in consequence:
  • Manipulating define constants without values, as this would require masking them by comments :-).

Thanks for listening!

With kind regards,
Andreas.


#6

Dear @clemens,

please have a look at the things we already prepared with “Use boolean define feature flags in the Open Hive firmware”. We would be very happy to merge these into mainline sooner than later after getting your acknowledgement, as you are the main author of this guy. We can assure a successful outcome as the examples above for compiling the Open Hive firmware are already using these amendments from the branch “node-gprs-http-firmware-builder”.

Thanks in advance for reviewing this change!

With kind regards,
Andreas.