This tracks the development progress of the firmware builder available as an API endpoint at https://swarm.hiveeyes.org/api.
Announcing Beta
We made the firmware builder subsystem actually work, mitigated some minor issues and added support for the Xtensa architecture used by ESP8266 chips from Espressif.
Introduction
As the firmware repository hosts different variants of firmwares, each with its unique feature set, there are different HTTP calls to the firmware builder endpoint obtaining different configuration parameters. Please read the source code to get an idea about the different parameters available at their respective endpoints.
AVR firmwares
Canonical Open Hive firmware: GPRS, HTTP, CSV
The battle-tested openhive-node-gprs.ino firmware by @clemens for Open Hive GSM Solar sensor nodes based on AVR MCUs.
http --timeout=300 --download POST \
https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
ref=node-gprs-http-firmware-builder update_submodules=false \
path=node-gprs-http makefile=Makefile-FWB.mk \
isScale=true isTemperatureArray=false
Mois Box firmware: YunBridge, HTTP, CSV
The node-yun.ino firmware powering the Mois sensor node by @mois.
http --timeout=300 --download POST \
https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
ref=master update_submodules=false \
path=node-yun-http makefile=Makefile-Linux.mk
Scale adjustment firmwares
Nomen est omen: Firmwares for load cell adjustment by @clemens.
For AVR/HX711
http --timeout=300 --download POST \
https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
ref=master update_submodules=false \
path=scale-adjust/HX711 makefile=Makefile-Linux.mk
For AVR/ADS1231
http --timeout=300 --download POST \
https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
ref=master update_submodules=false \
path=scale-adjust/ADS1231 makefile=Makefile-Linux.mk
AVR firmwares for ISM radio
Have a look at the generic firmware (config.h, generic.ino) by @einsiedlerkrebs for RFM69, RFM95/LoRa communication in a multi-node environment. The firmware can be configured to work in three different modes, which is very cool:
- Node: Obtains sensor readings from connected sensor hardware. Transmits data using BERadio over RFM69.
- Transceiver: Receives RFM69 packets and forwards them verbatim to RFM96/LoRa.
- Gateway: Receives radio packets over RFM96/LoRa and emits the payloads verbatim to its UART interface.
See also Next-generation sensor- and telemetry-node firmware for more details.
Role »node«: RFM69, BERadio
http --timeout=300 --download POST \
https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
ref=master update_submodules=false \
path=generic makefile=Makefile-Linux.mk \
BOARD_TAG=pro BOARD_SUB=8MHzatmega328 \
IS_NODE=true HE_BERadio=true HE_CONTAINERS=true \
HE_TEMPERATURE=true HE_HUMIDITY=true HE_SCALE=true
Role »transceiver«: RFM69, RFM96/LoRa
http --timeout=300 --download POST \
https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
ref=master update_submodules=false \
path=generic makefile=Makefile-Linux.mk \
IS_TRANSCEIVER=true
Role »gateway«: RFM96/LoRa, UART, MQTT
http --timeout=300 --download POST \
https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
ref=master update_submodules=false \
path=generic makefile=Makefile-Linux.mk \
IS_GATEWAY=true
Espressif / ESP8266 firmwares
Canonical Open Hive firmware: WiFi, HTTP, CSV
The battle-tested openhive-node-wifi.ino firmware for Open Hive WiFi Solar and similar sensor nodes by @clemens based on the ESP8266.
http --timeout=300 --download POST \
https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
ref=node-gprs-http-firmware-builder update_submodules=false \
path=node-gprs-http architecture=esp makefile=Makefile-ESP8266.mk \
isWifi=true WLAN_SSID=wifi-ssid-xxx WLAN_PW=wifi-password-yyy
Modern, compact firmware: WiFi, MQTT, JSON
The node-wifi-mqtt-homie.ino firmware powers the ESP8266 beehive scale based on Homie by @Alex and is based on a convenient development framework for ESP8266 called “Homie for ESP8266” following the Homie convention, which is around for some time but is still getting more and more popular. All reconfigurable parameters can be defined at runtime using the Homie configuration UI, a convenient captive portal html interface. See also Inbetriebnahme von node-wifi-mqtt-homie mit Hiveeyes Anbindung.
http --timeout=300 --download POST \
https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
ref=master update_submodules=false \
path=node-wifi-mqtt-homie architecture=esp makefile=Makefile
Modern, flexible firmware: WiFi, MQTT, JSON
Grown up from a simple educational firmware, node-wifi-mqtt.ino now is a maturing code base from a collaborative effort by @clemens, @karsten, @gtuveri, @Thias and @andreas. It is suitable for Open Hive WiFi Solar, Adafruit HUZZAH and other sensor nodes based on the ESP8266. Take a look at this piece if you want to get started and maybe even beyond: The code is really easy to reconfigure, tweak and extend. Almost all variables can be redefined using the firmware builder API interface with a naming scheme which is easy to follow.
http --timeout=300 --download POST \
https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.bin \
ref=master update_submodules=false \
path=node-wifi-mqtt architecture=esp makefile=Makefile \
WIFI_SSID_1=the-beekeepers \
WIFI_PASS_1=secret \
MEASUREMENT_INTERVAL="60 * 1000" \
DEEPSLEEP_ENABLED=true \
LOADCELL_ZERO_OFFSET=53623.0f \
LOADCELL_KG_DIVIDER=18053
Scale adjustment firmwares
Nomen est omen: Firmwares for load cell adjustment by @clemens.
For ESP8266/HX711
http --timeout=300 --download POST \
https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
ref=master update_submodules=false \
path=scale-adjust/HX711 architecture=esp makefile=Makefile-ESP8266.mk
For ESP8266/ADS1231
http --timeout=300 --download POST \
https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex \
ref=master update_submodules=false \
path=scale-adjust/ADS1231 architecture=esp makefile=Makefile-ESP8266.mk
Contributions are welcome!
As this firmware builder subsystem is still in the making, we would be happy to receive valuable feedback about the general idea and whether you will find it useful for producing custom firmwares without having any software environment installed on your machine.
We would be glad to hear if everything works as intended and even more if you could spot anomalies and errors or come across other kinds of edge cases we didn’t care about yet. All of your feedback will help to improve the robustness of the system.
There might even be a user interface around the corner… ;].
Cheers!
The Hiveeyes people.
1 Like
and whether you will find it useful for producing custom firmwares without having any software environment installed on your machine.
Many thanks Andreas! Is there a short step-by-step tutorial about this? Or does it only make sense as soon as the mentioned user interface is in place?
Regards,
Clemens
Hi Clemens!
You are welcome, thanks for your acknowledgement!
Thanks for asking! While developing this subsystem the other day, we wrote down some words about the general concept and how it is designed, please find them here:
However, there is currently no other kind of documentation yet which a) describes the single parameters in more detail or b) offers something in the style of a step-by-step tutorial. Go figure… :-)
Seriously, we were hoping to get this going as a collaborative effort between some of us who are interested in this topic over the course of the next days or weeks. Please don’t hesitate to share your thoughts, we would appreciate your help because it is important to us to make everyone feel comfortable with this feature. Feel free to ask any questions to bootstrap this process eventually, we will be happy to answer!
It will definitively be very cool to just head to a beautifully designed user interface, check some features, press the button and off you go! However, designing and building something like that requires a significant amount of effort to be put into.
That’s why we are announcing the infrastructural “Beta”: We want to encourage all of you to use what we already have to gain some experience and receive valuable feedback about how to iterate and improve on top of this. Especially after we now have tapped into building for the Espressif architecture as a second target after the AVR architecture and mitigated some other issues in the second iteration on the infrastructure machinery under the hood, we consider the subsystem to be almost ready for general consumption. As we have been able to successfully compile all of our representative firmwares listed above, we are confident everything works as designed in general.
There will definitively be quirks: Finding and mitigating them is what we are aiming at right now with this call to you.
Feel free to ask us anything, we are happy to answer.
Let’s get started…
About the firmware builder
The firmware builder subsystem receives a HTTP POST request describing the specifications (environment and features) of the firmware source code. The source code is amended according to these specifications and passed down to the cross compiler. When successful, the HTTP response contains a binary payload representing the “.hex” or “.bin” file to be uploaded to the MCU flash.
The format of the firmware builder HTTP endpoint is:
https://swarm.hiveeyes.org/api/hiveeyes/testdrive/area-42/node-1/firmware.hex
Please continue reading if you are interested in further details about the innards of this subsystem. It will make you more familiar with what is actually going on under the hood.
Goal
As a design goal, we wanted to achieve to make this work with almost no constraints on the source code and the Makefile itself: The variable replacement machinery should just look for “#define” macros and Makefile variables and manipulate them according to the input of the user running the unattended firmware builder. After the changes have been applied, the amended source code artifact is passed down to the build process and the resulting “.hex” or “.bin” file is handed back to the user.
Implementation
The machinery currently uses these firmware builder search patterns and the functions “patch_files”, “patch_file” and “find_variable”, see firmware builder replacement machine. It’s just a few lines of code and really easy to follow.
As this machinery isn’t made of any magic ingredients, it can achieve its power and robustness just under certain constraints.
Firmware source code guidelines
We prepared some words about what to care about on the firmware source code side. So this is a first guideline to follow about how to make your code work best together with the firmware builder infrastructure.
User configurable parameters
Just use regular “#define” directives to announce a parameter to be manipulated by the firmware builder. In other words: Every variable defined by “#define” directives can be amended. Cool, isn’t it?
Please have a look at “LOADCELL_ZERO_OFFSET” and “LOADCELL_KG_DIVIDER” definitions in node-wifi-mqtt.ino as a blueprint.
Feature flags
Please use “#define” constants with real boolean values as feature flags, like:
#define foobar true
#define bazqux false
instead of “#define foo” definitions and “#ifdef foo” / “#ifndef foo” checks on them. In our experience, this will even improve the readability and is easier to follow when reading the source code as a human - after it turned out it does help the machines already ;].
Please have a look at the sensor configuration of node-wifi-mqtt.ino as a blueprint.
Background / Reality check
It turned out there are minor constraints we have to put on the style of the “#define” macros - we already had a little conversation about this detail: It is difficult to get two specific things working in a robust manner:
- A conditional “enablement” of things masked by comments, e.g. “
//#define foo” - and in consequence: - Manipulating define constants without values, as this would require masking them by comments :-).
Thanks for listening!
With kind regards,
Andreas.
Dear @clemens,
please have a look at the things we already prepared with “Use boolean define feature flags in the Open Hive firmware”. We would be very happy to merge these into mainline sooner than later after getting your acknowledgement, as you are the main author of this guy. We can assure a successful outcome as the examples above for compiling the Open Hive firmware are already using these amendments from the branch “node-gprs-http-firmware-builder”.
Thanks in advance for reviewing this change!
With kind regards,
Andreas.