Stabilität und längere Testzeiträume des Terkin-Datenloggers

caro · August 7, 2019, 11:45am

Danke an alle Tester*innen! Ich habe zwei Bitten für zukünftige Tests und deren Dokumentation:
Könntet ihr bitte

die Softwareversion (hiveeyes / hiverize) mit Versionsnummer oder (falls nicht vorhanden) Pull-Datum sowie das verwendete Dateisystem (FatFS / littleFS) mit angeben
Elektronik nicht im Regen liegen lassen. Die Steckverbindungen mit den Pfostenbuchsen sind nicht wasserfest! Sehr gern stelle ich für Tests weitere Gehäuse zur Verfügung.
Danke!

Andreas · August 10, 2019, 12:05am

Im Rahmen der Untersuchungen zu elektromagnetischen Störungen des Wägezellen-Subsystems konnten wir noch die folgenden Beobachtungen zur Laufzeit feststellen.

Während der letzten Testreihe mit einem Vorseriengerät über ca. 170 Stunden ^[1] musste das Gerät zweimal per Reset oder Power-Cycle neu angekurbelt werden. Wir kennen die Ursache dafür noch nicht, der einzige Indikator war, dass das Blaulicht zu blinken aufhörte ^[2].
Da der Eumel ausschließlich an einem USB-Netzteil angeschlossen war, konnte die UART-Ausgabe leider nicht zur Laufzeit beobachtet werden, in der sich entsprechende Spuren hätten finden können.
Um diesen STOP-THE-WORLD-FOREVERs nachzugehen, hängt die Appliance nun an einem RaspberryPi3 SBC, um Stromversorgung plus dauerhaftes Debugging zu realisieren, ohne damit implizit meine Workstation über Tage an das Gerät fesseln zu müssen. Dieses Setup haben wir unter Monitoring and recording the serial interface output of a microcontroller attached to an UART interface näher beschrieben, um einen entsprechenden Nachbau einfacher zu machen.

Mit dem commit Revert "Adjust waiting time after resetting 1-Wire bus" · hiveeyes/hiveeyes-micropython-firmware@11e8eaf · GitHub vor ein paar Tagen waren hoffentlich endlich alle Sensor-Reading-Adjustments ordentlich im Kasten, so dass sich aus meiner Sicht diese Art von Testreihe überhaupt erstmalig lohnt. ↩︎
Ganz schön flashy übrigens, wenn das Teil in der Nacht in einem dunklen Zimmer ohne Lichtabschirmung steht und ungehindert seine Signalisierungen über die weißen Zimmerwände in den Innenhof reflektieren kann. ↩︎

Andreas · August 10, 2019, 4:21am

Direkt beim Starten des Geräts am SBC ist der Datenlogger zweimal sporadisch ausgerutscht.

   15.7152 [terkin.device            ] INFO   : Starting networking
   15.7978 [terkin.network.wifi      ] INFO   : WiFi STA: Networking address (MAC): {'ap_mac': '80:7d:3a:c3:42:bd', 'sta_mac': '80:7d:3a:c3:42:bc'}
   15.8128 [terkin.network.wifi      ] INFO   : WiFi STA: Networking address (IP):  ('0.0.0.0', '0.0.0.0', '0.0.0.0', '0.0.0.0')
Traceback (most recent call last):
  File "main.py", line 72, in <module>
  File "main.py", line 67, in main
  File "datalogger.py", line 138, in start
  File "device.py", line 70, in start_networking
  File "device.py", line 61, in start_networking
  File "network/core.py", line 35, in start_wifi
  File "network/wifi.py", line 52, in start
TimeoutError: Connection to AP Timeout!

   26.5656 [terkin.api.http          ] INFO   : Setting up HTTP API
Unhandled exception in thread started by <bound_method>
Traceback (most recent call last):
  File "network/ip.py", line 24, in start_real
OSError: Network card not available
   26.9306 [terkin.api.http          ] INFO   : Starting HTTP server
Traceback (most recent call last):
  File "main.py", line 72, in <module>
  File "main.py", line 67, in main
  File "datalogger.py", line 143, in start
  File "device.py", line 211, in start_network_services
  File "network/core.py", line 71, in start_httpserver
  File "api/http.py", line 70, in start
  File "microWebSrv.py", line 221, in Start
OSError: Network card not available

Andreas · August 14, 2019, 7:19pm

Zwischenbericht

Nach 420.330,4291 Sekunden, also gut 4 Tagen und 20 Stunden, hat der Datenlogger von sich aus neugestartet und ist wohl nach dem Neustart auf dem gleichen Fehler OSError: Network card not available ausgerutscht, der oben schon erwähnt wurde. Hier muss dringend nachgebessert werden.

Als Ursache für das Ausrutschen konnte diesmal dank Monitoring and recording the serial interface output of a microcontroller attached to an UART interface ein CORE DUMP beobachtet werden. Anbei finden sich entsprechende Logs dazu.

01-post-mortem-01-narrow.log (4.3 KB)
01-post-mortem-02-medium.log (42.5 KB)

STGTFO

   24.4832 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 1 retries left
   25.3060 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 0 retries left
   26.1432 [terkin.network.wifi      ] ERROR  : WiFi STA: Connecting to "GartenNetzwerk" failed
Traceback (most recent call last):
  File "network/wifi.py", line 117, in connect_stations
  File "network/wifi.py", line 192, in connect_station
WiFiException: WiFi STA: Unable to connect to "GartenNetzwerk"

   26.5553 [terkin.api.http          ] INFO   : Setting up HTTP API
Unhandled exception in thread started by <bound_method>
Traceback (most recent call last):
  File "network/ip.py", line 24, in start_real
OSError: Network card not available
   26.9223 [terkin.api.http          ] INFO   : Starting HTTP server
Traceback (most recent call last):
  File "main.py", line 72, in <module>
  File "main.py", line 67, in main
  File "datalogger.py", line 143, in start
  File "device.py", line 211, in start_network_services
  File "network/core.py", line 71, in start_httpserver
  File "api/http.py", line 70, in start
  File "microWebSrv.py", line 221, in Start
OSError: Network card not available

Überlegung

Für die Ursache des sporadischen CORE DUMPs könnten “weiche” Fehler sein, die über Single-Event Upsets (SEUs) zustande kommen können ^[1].

Zwischenfazit

Die exakte Ursache solcher sporadischer Abstürze muss uns an dieser Stelle jedoch gar nicht primär interessieren, wir müssen nur durch optimale Vorbereitung das Chaos gut im Griff haben.

Dankenswerterweise übernimmt der Watchdog-Timer (WDT) an dieser Stelle eine wichtige Aufgabe und führt das Gerät automatisch wieder aus dem Fehlerzustand heraus.

Ein Single Event Upset (SEU) ist ein Soft Error (deutsch „weicher“ Fehler), der beispielsweise beim Durchgang hochenergetischer ionisierender Teilchen durch Halbleiterbauelemente hervorgerufen werden kann. Der Fehler äußert sich beispielsweise als bitflip, also der Änderung des Zustandes eines einzelnen oder mehrerer Bits in Speicherbausteinen oder CPU-Registern, was wiederum zu einer Fehlfunktion des betroffenen Bauteils führen kann. Wissenschaftliches und gleichermaßen Kurioses dazu haben wir auch unter Soft errors caused by single-event upsets (SEUs) zusammengetragen. ↩︎

Andreas · August 15, 2019, 12:41pm

8 posts were merged into an existing topic: Lagerkoller wegen kosmischer Strahlung

Andreas · August 15, 2019, 1:46am

Verhalten bei der WiFi-Verbindung

Nach dem Wechsel von einer AVM FRITZ!Box 7320 auf eine AVM FRITZ!Box 7520 als DSL-Modem und WiFi Access Point konnte ich ein paar Verhaltensänderungen bei der Verbindungsaufname des FiPy zum WiFi Access Point feststellen.

Im Gegensatz dazu, dass vorher bei jedem Neustart die IP-Adresse wechselte, bleibt sie nun stabil. Möglicherweise ist also der DHCP-Server auf der Box irgendwie besser geworden.
Bisher dauerte die Verbindungsaufname ca. fünf Sekunden, nun klappt es oft schon nach drei Sekunden.
Allerdings klappt die Verbindung zum AP deterministisch nur jedes zweite Mal, also in der Hälfte aller Fälle nicht. Im Anhang kann man die Verläufe im Erfolgsfall und im Fehlerfall nachlesen.

Anhang

STA connection succeeded

   18.0049 [terkin.device            ] INFO   : Starting networking
   18.1074 [terkin.network.wifi      ] INFO   : WiFi STA: Networking address (MAC): {'ap_mac': '80:7d:3a:c2:de:45', 'sta_mac': '80:7d:3a:c2:de:44'}
   18.1239 [terkin.network.wifi      ] INFO   : WiFi STA: Networking address (IP):  ('0.0.0.0', '0.0.0.0', '0.0.0.0', '0.0.0.0')
   18.1572 [terkin.network.wifi      ] INFO   : WiFi STA+AP: Starting interface
   18.1763 [terkin.network.wifi      ] INFO   : WiFi STA: Directly connecting to configured networks: ['GartenNetzwerk']
   18.1966 [terkin.network.wifi      ] INFO   : WiFi STA: Prepare connecting to network "GartenNetzwerk"
   18.2700 [terkin.network.wifi      ] INFO   : WiFi STA: Auth mode from NVRAM with key=wa.0a10c7c77510, value=3
   18.2906 [terkin.network.wifi      ] INFO   : WiFi STA: Attempt connecting to network "GartenNetzwerk" with auth mode "3"
   18.3113 [terkin.network.wifi      ] INFO   : WiFi STA: Starting connection to "GartenNetzwerk" with timeout of 15.0 seconds
   18.3346 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 17 retries left
   19.1591 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 16 retries left
   19.9851 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 15 retries left
   20.8110 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 14 retries left
   21.6484 [terkin.network.wifi      ] INFO   : WiFi STA: Connected to "('GartenNetzwerk', 'fipy-wlan-de44')" with IP address "192.168.178.30"
   21.7385 [terkin.network.wifi      ] INFO   : WiFi STA: Networking address (MAC): {'ap_mac': '80:7d:3a:c2:de:45', 'sta_mac': '80:7d:3a:c2:de:44'}
   21.7543 [terkin.network.wifi      ] INFO   : WiFi STA: Networking address (IP):  ('192.168.178.30', '255.255.255.0', '192.168.178.1', '192.168.178.1')
   21.7756 [terkin.network.core      ] INFO   : Network interface ready

STA connection failed

   18.0040 [terkin.device            ] INFO   : Starting networking
   18.1067 [terkin.network.wifi      ] INFO   : WiFi STA: Networking address (MAC): {'ap_mac': '80:7d:3a:c2:de:45', 'sta_mac': '80:7d:3a:c2:de:44'}
   18.1233 [terkin.network.wifi      ] INFO   : WiFi STA: Networking address (IP):  ('0.0.0.0', '0.0.0.0', '0.0.0.0', '0.0.0.0')
   18.1566 [terkin.network.wifi      ] INFO   : WiFi STA+AP: Starting interface
   18.1758 [terkin.network.wifi      ] INFO   : WiFi STA: Directly connecting to configured networks: ['GartenNetzwerk']
   18.1959 [terkin.network.wifi      ] INFO   : WiFi STA: Prepare connecting to network "GartenNetzwerk"
   18.2694 [terkin.network.wifi      ] INFO   : WiFi STA: Auth mode from NVRAM with key=wa.0a10c7c77510, value=3
   18.2900 [terkin.network.wifi      ] INFO   : WiFi STA: Attempt connecting to network "GartenNetzwerk" with auth mode "3"
   18.3106 [terkin.network.wifi      ] INFO   : WiFi STA: Starting connection to "GartenNetzwerk" with timeout of 15.0 seconds
   18.3339 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 17 retries left
   19.1575 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 16 retries left
   19.9845 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 15 retries left
   20.8077 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 14 retries left
   21.6305 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 13 retries left
   22.4536 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 12 retries left
   23.2766 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 11 retries left
   24.0996 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 10 retries left
   24.9215 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 9 retries left
   25.7436 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 8 retries left
   26.5656 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 7 retries left
   27.3875 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 6 retries left
   28.2095 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 5 retries left
   29.0316 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 4 retries left
   29.8535 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 3 retries left
   30.6756 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 2 retries left
   31.4969 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 1 retries left
   32.3185 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 0 retries left
   33.1520 [terkin.network.wifi      ] ERROR  : WiFi STA: Connecting to "GartenNetzwerk" failed
Traceback (most recent call last):
  File "/flash/lib/terkin/network/wifi.py", line 126, in connect_stations
  File "/flash/lib/terkin/network/wifi.py", line 201, in connect_station
WiFiException: WiFi STA: Unable to connect to "GartenNetzwerk"

   33.1909 [terkin.network.wifi      ] INFO   : WiFi STA: Forgetting NVRAM data for network "GartenNetzwerk"
   33.2824 [terkin.network.wifi      ] ERROR  : WiFi STA: Connecting to any network candidate failed. Please check your WiFi configuration for one of the station candidates 1.
   33.3003 [terkin.network.wifi      ] WARNING: Todo: We might want to switch to AP mode here or alternatively buffer telemetry data to flash to be scheduled for transmission later.
   33.3226 [terkin.network.core      ] INFO   : Network interface ready

wtf · August 15, 2019, 7:15am

Du schreibst weiter unten, dass Dein (neues) WLAN ohnehin jeden zweiten Verbindungsversuch zum Scheitern verurteilt. Wie unterscheidet sich der Umgang mit dem Fehler “unable to connect” dort wie hier?

Andreas · August 15, 2019, 12:49pm

Scheinbar, leider, ja.

Da wir noch keinen Hintergrund-Dienst zur Überwachung der Verbindung im WiFi-Manager haben, habe ich versucht, die Dinge erst einmal pragmatisch zu verbessern. Ein paar neue commits ^[1] tragen dazu bei, dass a) der Datenlogger an den genannten Stellen nun nicht mehr unmotiviert abstürzt und b) dass zwei Verbindungsversuche unternommen werden weil ja immer einer von beiden deterministisch fehlschlägt.

Die nächsten Ausbaustufen ^[2] kommen dann hoffentlich bald in einer späteren Iteration.

Comparing 495890c085fe459fb1fcc6112875d5fc77abc58a...6eddea24b3b4ee8ba9154d38de0de0c7aad905a2 · hiveeyes/terkin-datalogger · GitHub ↩︎
Pragmatischer Service-Thread zur Verbindungsüberwachung oder ggf. ein Ruf zu ensure_connection() kurz vor dem Telemetrie-Akt. Generell wollen wir die bereits separat getrennten Domänen (Sensor vs. Telemetrie) gerne auch beizeiten parallelisieren, um Effizienzgewinne zu erzielen. Momentan wird noch so ziemlich alles sequentiell abgearbeitet. ↩︎

Andreas · August 15, 2019, 12:59pm

2 posts were merged into an existing topic: Lagerkoller wegen kosmischer Strahlung

Andreas · August 15, 2019, 8:37pm

Beobachtungen

In der Praxis sieht das dann so aus wie unten im Log bei Connect to WiFi twice zu sehen. Erst einmal schlägt die Verbindung fehl aber nachdem noch einmal nachgehakt wurde, klappt es erfolgreich. Alles andere als optimal, ist halt aber scheinbar so.

Connect to WiFi twice shmoo

=============================
Bee Observer Datalogger 0.5.1
=============================
CPU freq     160.0 MHz
Device id    807d3ac342bc

Python  : 3.4.0
lorawan : 1.0.2
machine : FiPy with ESP32
nodename: FiPy
release : 1.20.0.rc12.1
sigfox  : 1.0.1
sysname : FiPy
version : 6c0000f on 2019-08-01

   23.7123 [terkin.device            ] INFO   : Starting networking
   23.8151 [terkin.network.wifi      ] INFO   : WiFi STA: Networking address (MAC): {'ap_mac': '80:7d:3a:c3:42:bd', 'sta_mac': '80:7d:3a:c3:42:bc'}
   23.8323 [terkin.network.wifi      ] INFO   : WiFi STA: Networking address (IP):  ('0.0.0.0', '0.0.0.0', '0.0.0.0', '0.0.0.0')
   23.8654 [terkin.network.wifi      ] INFO   : WiFi STA+AP: Starting interface
   23.8867 [terkin.network.wifi      ] INFO   : WiFi STA: Connecting to configured networks: ['GartenNetzwerk']. Attempt: #1
   23.9091 [terkin.network.wifi      ] INFO   : WiFi STA: Prepare connecting to network "GartenNetzwerk"
   23.9923 [terkin.network.wifi      ] INFO   : WiFi STA: Auth mode from NVRAM with key=wa.0a10c7c77510, value=3
   24.0140 [terkin.network.wifi      ] INFO   : WiFi STA: Attempt connecting to network "GartenNetzwerk" with auth mode "3"
   24.0376 [terkin.network.wifi      ] INFO   : WiFi STA: Starting connection to "GartenNetzwerk" with timeout of 15.0 seconds
   24.0640 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 17 retries left
   24.8910 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 16 retries left
   25.7192 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 15 retries left
   26.5462 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 14 retries left
   27.3736 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 13 retries left
   28.2000 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 12 retries left
   29.0255 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 11 retries left
   29.8515 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 10 retries left
   30.6755 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 9 retries left
   31.4993 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 8 retries left
   32.3224 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 7 retries left
   33.1464 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 6 retries left
   33.9704 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 5 retries left
   34.7944 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 4 retries left
   35.6183 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 3 retries left
   36.4423 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 2 retries left
   37.2665 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 1 retries left
   38.0903 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 0 retries left
   38.9280 [terkin.network.wifi      ] ERROR  : WiFi STA: Connecting to "GartenNetzwerk" failed
Traceback (most recent call last):
  File "/flash/lib/terkin/network/wifi.py", line 128, in connect_stations
  File "/flash/lib/terkin/network/wifi.py", line 203, in connect_station
WiFiException: WiFi STA: Unable to connect to "GartenNetzwerk"

   38.9691 [terkin.network.wifi      ] INFO   : WiFi STA: Forgetting NVRAM data for network "GartenNetzwerk"
   39.0721 [terkin.network.wifi      ] ERROR  : WiFi STA: Connecting to any network candidate failed. Please check your WiFi configuration for one of the station candidates 1.
   39.0921 [terkin.network.wifi      ] WARNING: Todo: We might want to buffer telemetry data to flash memory to be scheduled for transmission later.
   39.1095 [terkin.network.wifi      ] INFO   : WiFi STA: Connecting to configured networks: ['GartenNetzwerk']. Attempt: #2
   39.1325 [terkin.network.wifi      ] INFO   : WiFi STA: Prepare connecting to network "GartenNetzwerk"
   39.2188 [terkin.network.wifi      ] INFO   : WiFi STA: Unknown auth mode for network "GartenNetzwerk", invoking WiFi scan
   39.2358 [terkin.network.wifi      ] INFO   : WiFi STA: Scanning for networks
   42.2570 [terkin.network.wifi      ] INFO   : WiFi STA: Networks available: ['REDACTED', 'REDACTED', 'YADDA', 'GartenNetzwerk', 'REDACTED', 'REDACTED']
   42.3047 [terkin.network.wifi      ] INFO   : WiFi STA: Storing auth mode into NVRAM with key=wa.0a10c7c77510, value=3
   42.3286 [terkin.network.wifi      ] INFO   : WiFi STA: Attempt connecting to network "GartenNetzwerk" with auth mode "3"
   42.3546 [terkin.network.wifi      ] INFO   : WiFi STA: Starting connection to "GartenNetzwerk" with timeout of 15.0 seconds
   42.3840 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 17 retries left
   43.2122 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 16 retries left
   44.0407 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network "GartenNetzwerk" to come up, 15 retries left
   44.8859 [terkin.network.wifi      ] INFO   : WiFi STA: Connected to "('GartenNetzwerk', 'fipy-wlan-42bc')" with IP address "192.168.178.21"
   44.9752 [terkin.network.wifi      ] INFO   : WiFi STA: Networking address (MAC): {'ap_mac': '80:7d:3a:c3:42:bd', 'sta_mac': '80:7d:3a:c3:42:bc'}
   44.9933 [terkin.network.wifi      ] INFO   : WiFi STA: Networking address (IP):  ('192.168.178.21', '255.255.255.0', '192.168.178.1', '192.168.178.1')
   45.0158 [terkin.network.core      ] INFO   : Network interface ready
   45.0446 [terkin.device            ] INFO   : [LoRa] Disabling LoRa interface as no antenna has been attached. ATTENTION: Running LoRa without antenna will wreck your device.
   45.0660 [terkin.device            ] INFO   : Starting telemetry

Weitere Nachforschungen

Für die Ursachenforschung könnte man noch ein paarmal den Mond umkreisen. Vermutlich ist es so, dass der halbgare Zustand durch den WiFi-Scan gerettet wird, der zwangsläufig beim zweiten Versuch folgt ^[1].

Einschätzung des Problems

Alles kein Drama. Dass die Verbindung mal nicht klappt, passierte innerhalb der letzten Testreihe nur zwei Mal innerhalb von 18 Stunden Laufzeit im Produktionsmodus (mit Deep Sleep, mit Watchdog, ohne LED-Heartbeat; dafür aber mit Logging und am Strom ohne Batterie) und wird ja nun durch den kleinen Workaround kompensiert.

root@kotori-one:~# cat /home/pi/sermon-workbench-fipy.log | grep Exception

WiFiException: WiFi STA: Unable to connect to "GartenNetzwerk"
WiFiException: WiFi STA: Unable to connect to "GartenNetzwerk"

P.S.: Wenn man das Log genau anschaut, entdeckt man so eine Versionsnummer:

Das kommt aus den ersten Versuchen zu Module freezing for Pycom MicroPython - #5 by Andreas.

… nachdem die auth-mode Informationen gelöscht wurden, die sich zwischenzeitlich im NVRAM gemerkt wurden, damit man im Normalfall keinen Scan benötigt, um das WiFi anzufahren. ↩︎

Andreas · August 15, 2019, 8:59pm

… ist natürlich viel zu unpräzise. Das Schlafintervall lag hier bei 60 Sekunden, zusammen mit den 30 Sekunden Laufzeit ergeben sich über die 18 Stunden Laufzeit also ungefähr 720 Zyklen, von denen wiederum zwei fehlschlugen.

wtf · August 17, 2019, 11:04am

Wie kommt das Terkin denn auf die AP-MAC-Adresse? Ist das ne Info, die nach nem Befehl “verbinde Dich mit $ESSID und $Passwort” generiert wird, oder ist diese “MAC-Adresse” (BSSID präzise zu sein) irgendwie hard-coded und damit Bestandteil des “verbinde Dich”-Befehls?

Also zunächst: Stimmt die denn überhaupt? Ich kann mir schwerlichst vorstellen, dass sowohl die MAC-Adresse des FiPy-Interfaces als auch die Deines WLAN-APs (mutmaßlich AVM Fritzbox) beide vom Vendor Expressif Inc. stammen und direkt nacheinander vergeben wurde (increase of one between ap_mac and sta_mac).

Andreas · August 17, 2019, 2:16pm

Hey,

danke fürs Lesen der Log-Meldungen ^[1].

Alles ist ganz einfach: Beide dort angegebenen MAC-Adressen sind im FiPy einprogrammiert. Die eine ist die designierte MAC-Adresse, die für die WiFi-Netzwerkkarte im STA-Modus benutzt wird (als WiFi-Client), die andere ist die, die im AP-Modus benutzt wird (als WiFi-AP).

Seit einiger Zeit schalten wir pauschal in der Terkin-Firmware beide Modi an, das kann sich zukünftig bestimmt noch einmal ändern, wenn wir den AP-Mode nur unter bestimmten Umständen (auf Zuruf per Knopfdruck oder Steuerbefehl) aktivieren.

Die MAC-Adresse des STA-Interfaces korreliert 1:1 mit der Geräte-ID.

Device id    807d3ac342bc

In der Tat wird für die Vergabe der MAC-Adresse des AP-Interfaces einfach eins auf die MAC-Adresse des STA-Interfaces draufaddiert, ja.

Hilft das weiter, um an dieser Stelle mehr Licht ins Dunkle zu bringen?

You know who you are. ↩︎

Andreas · August 19, 2019, 1:22am

Per Parallelize networking subsystem · hiveeyes/terkin-datalogger@22ed127 · GitHub konnten wir die Infrastruktur nun endlich auch in diesem Bereich verbessern.

clemens · August 19, 2019, 8:54am

Magst du mal erklären, was da nun parallel läuft, wir brauchen ja zuerst die Sensordaten um sie dann erst zu schicken. Was bringt das ca. in Sekunden für die runtime?

Andreas · August 19, 2019, 5:01pm

Auf die Gesamtlaufzeit wirkt sich das kaum aus. “Einmal hochfahren und messen” dauert derzeit immer noch um die 30 Sekunden lang.

Wichtig ist an dieser Stelle weiterhin die Stabilität bzw. Robustheit: Mit nun im Hintergrund dauerhaft wirksamen Servicing des WiFi-Moduls ist die Appliance gegenüber Verbindungsstörungen bei der Konnektivität zum Access Point nun auch im Live Mode resilienter.

Andreas · August 28, 2019, 11:16am

WiFi connectivity woes

Wir konnten bei einigen Beobachtungen feststellen, dass folgende WiFi-Metriken rausgegeben werden, wenn die Verbindung nicht klappt.

system.wifi.bandwidth                            2
system.wifi.channel                              6
system.wifi.country                             CN
system.wifi.max_tx_power                        78

Wenn die Verbindung klappt, wurden folgende Metriken herausgegeben.

system.wifi.bandwidth                            2
system.wifi.channel                              1
system.wifi.country                             DE
system.wifi.max_tx_power                        78
system.wifi.rssi                               -52

clemens · August 29, 2019, 7:01am

Das Auffälligste ist, dass system.wifi.country sich ändert, ich habe allerdings auch CN bei erfolgreichen Übertragungen (in Deutschland ;-) stehen, s. https://swarm.hiveeyes.org/api/hiveeyes/statista/hds/hive-1/data.txt?from=now-30m&include=system.wifi.bandwidth,system.wifi.channel,system.wifi.country,system.wifi.max_tx_power,system.wifi.rssi

clemens · August 29, 2019, 8:03am

Weil ich gerade in meinen Daten gesehen haben, dass system.wifi.rssi nicht immer übertagen wurde, ein kleiner Zwischenstand von einem System, das ich Anfang der Woche installiert habe:

In den letzten 2 Tagen wurden ca. 1700 Datensätze übertragen, das sind alle 1:40 ein Datensatz. Mit einem konfigurierten measurement interval von 60 Sekunden ergibt sich mit aktiviertem deep sleep eine runtime von 40 Sekunden je cycle.

Bei den DS18B20 gab es zwischen 7 und 9% missings, s.o., bei den system.wifi.rssi gab es 3 % nicht übertragene Werte im Datensatz, was mir bisher noch nicht aufgefallen ist.

Alle anderen Sensoren, sowohl die System-internen als auch Waage und BME hatten keinerlei Ausfälle!

export-hive-1_2018-08-27to29.xlsx (270.0 KB)

Andreas · September 5, 2019, 10:45pm

Langzeittestreihe Gamma

Wir starteten am Abend des 27. August 2019 eine Meßreihe für erste Langzeittests, die weitestgehend unbeaufsichtigt bis in die Nacht des 5. September 2019 reichte. Ein paar Ereignisse und Beobachtungen dazu haben wir im Dashboard annotiert.

https://swarm.hiveeyes.org/grafana/d/0CJxruMZz/amo-fipy-workbench?from=20190827T000000&to=20190905T230000

Softwarestand

=====================================
Hiveeyes MicroPython Datalogger 0.6.0
=====================================
CPU freq     160.0 MHz
Device id    807d3ac342bc

Python  : 3.4.0
lorawan : 1.0.2
machine : FiPy with ESP32
nodename: FiPy
release : 1.20.0.rc12.1
sigfox  : 1.0.1
sysname : FiPy
version : 6c0000f on 2019-08-01

Konfiguration

Stromversorgung und UART-Verbindung über RaspberryPi, siehe Monitoring and recording the serial interface output of a microcontroller attached to an UART interface.
60 Sekunden Pseudozyklus
Deep Sleep
Watchdog

Ergebnisse

Die Datenlogger-Firmware hat mittlerweile eine annehmbare Stabilität und Meßqualität erreicht. Herzlichen Dank an alle, die dabei mitgeholfen haben!

Beobachtungen

Während der Laufzeit traten sporadische Fehler auf, die das Gerät jedoch niemals zum Totalabsturz / Freeze brachten. Zum Beispiel handelte es sich um Dinge wie

Konnektivität fehlt komplett

Falls die Netzwerk-Konnektivität nicht rechtzeitig hergestellt werden konnte, bevor die MCU wieder schlafengelegt wird, schlägt die Telemetrie fehl. Dieser Zustand trat während der gesamten Laufzeit 11 mal ein:

$ cat 2019-09-06_sermon-workbench-fipy.log | grep ERROR | grep EAI_FAIL | wc -l
11

Beispiel

Insgesamt gab es hier 219 mal Probleme.

$ cat 2019-09-06_sermon-workbench-fipy.log | grep WARNING | grep DS18X20 | wc -l
219

   37.0152 [terkin.telemetry         ] INFO   : Connecting to MQTT broker at swarm.hiveeyes.org with username None
   37.0576 [terkin.telemetry         ] ERROR  : Connecting to MQTT broker at swarm.hiveeyes.org failed: OSError: [Errno 202] EAI_FAIL
Traceback (most recent call last):
  File "telemetry.py", line 515, in connect
  File "mqtt.py", line 26, in __init__
OSError: [Errno 202] EAI_FAIL

   37.1170 [terkin.telemetry         ] ERROR  : Telemetry to mqtt://swarm.hiveeyes.org/hiveeyes/testdrive/area-38/fipy-workbench-01 failed
Traceback (most recent call last):
  File "telemetry.py", line 119, in transmit
  File "telemetry.py", line 279, in transmit
  File "telemetry.py", line 460, in send
  File "telemetry.py", line 460, in send
TelemetryTransportError: Protocol adapter not connected: TelemetryAdapterError: Connecting to MQTT broker at swarm.hiveeyes.org failed: OSError: [Errno 202] EAI_FAIL

   37.2762 [terkin.telemetry         ] INFO   : Sending HTTP request to https://bee-observer.org/api/sensors
   37.2936 [terkin.telemetry         ] INFO   : Payload:     {"weight": 1.462, "t_i_1": 27.9375, "t": 26.77627, "t_i_2": 24.25, "key": "XgLPp1gLDM4JhZeq", "h": 52.95385, "p": 1017.906, "t_i_5": 35.56522}
   37.6553 [terkin.telemetry         ] ERROR  : Telemetry to https://bee-observer.org/api/sensors failed
Traceback (most recent call last):
  File "telemetry.py", line 119, in transmit
  File "telemetry.py", line 279, in transmit
  File "telemetry.py", line 337, in send
  File "urequests/__init__.py", line 144, in post
  File "urequests/__init__.py", line 60, in request
OSError: [Errno 202] EAI_FAIL

   37.7081 [terkin.datalogger        ] WARNING: Telemetry status: FAILURE. 2 out of 2 targets failed. Status: {'mqtt://swarm.hiveeyes.org/hiveeyes/testdrive/area-38/fipy-workbench-01': False, 'https://bee-observer.org/api/sensors': False}
   37.8192 [terkin.network.wifi      ] INFO   : WiFi STA: Waiting for network to come up within 2.742 seconds

Backend offline

Manchmal ist ein Backend nicht erreichbar. Das sieht in der Praxis folgendermaßen aus.

Beispiel

   35.5541 [terkin.telemetry         ] INFO   : Connecting to MQTT broker at swarm.hiveeyes.org with username None
   53.8956 [terkin.telemetry         ] ERROR  : Connecting to MQTT broker at swarm.hiveeyes.org failed: OSError: [Errno 113] ECONNABORTED
Traceback (most recent call last):
  File "telemetry.py", line 517, in connect
  File "mqtt.py", line 66, in connect
OSError: [Errno 113] ECONNABORTED

   53.9556 [terkin.telemetry         ] ERROR  : Telemetry to mqtt://swarm.hiveeyes.org/hiveeyes/testdrive/area-38/fipy-workbench-01 failed
Traceback (most recent call last):
  File "telemetry.py", line 119, in transmit
  File "telemetry.py", line 279, in transmit
  File "telemetry.py", line 460, in send
  File "telemetry.py", line 460, in send
TelemetryTransportError: Protocol adapter not connected: TelemetryAdapterError: Connecting to MQTT broker at swarm.hiveeyes.org failed: OSError: [Errno 113] ECONNABORTED

   54.0645 [terkin.telemetry         ] INFO   : Sending HTTP request to https://bee-observer.org/api/sensors
   54.0817 [terkin.telemetry         ] INFO   : Payload:     {"weight": 1.492, "t_i_1": 23.0625, "t": 23.37688, "t_i_2": 22.25, "key": "XgLPp1gLDM4JhZeq", "h": 39.42031, "p": 1017.708, "t_i_5": 33.65218}
   56.8377 [terkin.datalogger        ] WARNING: Telemetry status: FAILURE. 1 out of 2 targets failed. Status: {'mqtt://swarm.hiveeyes.org/hiveeyes/testdrive/area-38/fipy-workbench-01': False, 'https://bee-observer.org/api/sensors': True}

Modem kann nicht heruntergefahren werden

Dieser Zustand trat 280 mal ein.

$ cat 2019-09-06_sermon-workbench-fipy.log | grep 'Shutting down LTE modem failed' | wc -l
280

Beispiel

   20.2014 [terkin.device            ] ERROR  : Shutting down LTE modem failed
Traceback (most recent call last):
  File "device.py", line 282, in power_off_lte_modem
OSError: the requested operation failed

DS18B20 konnten nicht gelesen werden

Dieses Problem ist bereits bekannt und sollte noch per Retry reading of failed DS18B20 sensors verbessert werden.

Beispiel

Log-Auszug mit Problemen bei einem oder mehreren DS18B20 Sensoren.

   34.0366 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0396 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0369 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9655 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0334 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9923 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9705 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0314 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0240 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9661 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.7696 [hiveeyes.sensor_ds18x20  ] WARNING: No data from any DS18X20 devices on bus "onewire:0"
   33.9169 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   35.1140 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0547 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9716 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   35.1069 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0440 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0407 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0602 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9699 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9655 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.7646 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9580 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9174 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.8151 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9815 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0485 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0500 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   31.4821 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0406 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9952 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9595 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9536 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0520 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0506 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0548 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0294 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9750 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.9115 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9805 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0331 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0560 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0326 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0500 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9729 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0042 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0637 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0584 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0398 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0409 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9292 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9665 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9492 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9217 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9773 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9717 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.8333 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9519 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9718 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0558 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9631 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9507 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0385 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9685 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0316 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0439 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9688 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0359 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   31.3748 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0308 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9700 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0435 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9605 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9721 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0319 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9778 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0401 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0364 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0479 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9712 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0555 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9445 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0306 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0243 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0173 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.6904 [hiveeyes.sensor_ds18x20  ] WARNING: No data from any DS18X20 devices on bus "onewire:0"
   33.8798 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9457 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.8704 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9535 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9496 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.8768 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9821 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9832 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9404 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.6997 [hiveeyes.sensor_ds18x20  ] WARNING: No data from any DS18X20 devices on bus "onewire:0"
   34.0303 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0208 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9925 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9395 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   31.0679 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0061 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9287 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   30.9916 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.8661 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.7095 [hiveeyes.sensor_ds18x20  ] WARNING: No data from any DS18X20 devices on bus "onewire:0"
   33.9317 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.7154 [hiveeyes.sensor_ds18x20  ] WARNING: No data from any DS18X20 devices on bus "onewire:0"
   33.9446 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0040 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0063 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   31.5988 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.8644 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0239 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9083 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9249 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0013 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.8651 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0152 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.8841 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9617 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0059 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9515 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9285 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0168 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.7256 [hiveeyes.sensor_ds18x20  ] WARNING: No data from any DS18X20 devices on bus "onewire:0"
   33.9338 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9996 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.8969 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.8952 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9916 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9747 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9086 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9391 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9551 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9281 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.6879 [hiveeyes.sensor_ds18x20  ] WARNING: No data from any DS18X20 devices on bus "onewire:0"
   34.0099 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.8839 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0230 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0142 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0005 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9360 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9340 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0012 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.7214 [hiveeyes.sensor_ds18x20  ] WARNING: No data from any DS18X20 devices on bus "onewire:0"
   33.9538 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0192 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0237 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0188 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9529 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9992 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.7012 [hiveeyes.sensor_ds18x20  ] WARNING: No data from any DS18X20 devices on bus "onewire:0"
   33.9434 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9048 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0072 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9824 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.7115 [hiveeyes.sensor_ds18x20  ] WARNING: No data from any DS18X20 devices on bus "onewire:0"
   33.9951 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.6737 [hiveeyes.sensor_ds18x20  ] WARNING: No data from any DS18X20 devices on bus "onewire:0"
   34.0168 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0114 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9495 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9329 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.8918 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0052 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.8555 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0113 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9789 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9938 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9168 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0183 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0335 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9200 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9910 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.7119 [hiveeyes.sensor_ds18x20  ] WARNING: No data from any DS18X20 devices on bus "onewire:0"
   33.7054 [hiveeyes.sensor_ds18x20  ] WARNING: No data from any DS18X20 devices on bus "onewire:0"
   33.9071 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9457 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0119 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9536 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9487 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9263 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.7220 [hiveeyes.sensor_ds18x20  ] WARNING: No data from any DS18X20 devices on bus "onewire:0"
   33.7219 [hiveeyes.sensor_ds18x20  ] WARNING: No data from any DS18X20 devices on bus "onewire:0"
   34.0220 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0133 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9453 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.7007 [hiveeyes.sensor_ds18x20  ] WARNING: No data from any DS18X20 devices on bus "onewire:0"
   33.9122 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9988 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9121 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9260 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9413 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9365 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.7435 [hiveeyes.sensor_ds18x20  ] WARNING: No data from any DS18X20 devices on bus "onewire:0"
   33.7179 [hiveeyes.sensor_ds18x20  ] WARNING: No data from any DS18X20 devices on bus "onewire:0"
   33.8986 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0142 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9527 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.9918 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9948 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.8901 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0022 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9499 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9427 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9860 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.7531 [hiveeyes.sensor_ds18x20  ] WARNING: No data from any DS18X20 devices on bus "onewire:0"
   33.9688 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   35.0839 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9113 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9253 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   34.0236 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0292 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   34.0245 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9311 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"
   33.9272 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9786 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fc3944f"
   33.9439 [hiveeyes.sensor_ds18x20  ] WARNING: No response from DS18X20 device "28ff641d8fdf18c1"