Zwischenpufferung von Daten und Log-Informationen im RTC slow memory des ESP32

Andreas · July 28, 2019, 1:50pm

Die einzigen beiden Referenzen auf RTC_DATA_ATTR im Pycom Universum finden sich bei:

Andreas · July 28, 2019, 2:31pm

State of the onion with RTC slow memory on ESP32 from Pycom MicroPython

2016

Battery conservation techniques | Pycom user forum

2017

2018

2019

Bottom line

wtf · July 30, 2019, 3:57pm

Trüffel-schnüffel … das hier riecht interessant:

Scheint ne machine-imlementation in C von nem Individualisten zu sein, aus der Doku:

rtc.write(pos, value)

Write integer (32-bit) value to the position pos in RTC memory.
Return True on success, False if failed.

rtc.read(pos)

Read integer (32-bit) from the position pos in RTC memory.
Returns None if no value has not been written to the RTC integer memory yet or the RTC memory was corrupted (bad CRC), otherwise returns the integer written to the position or 0 (default value).

rtc.write_string(text)

Write the string text to RTC memory.
Return True on success, False if failed.

rtc.read_string()

Read the string from RTC memory.
Returns None if no string has been written to the RTC string memory yet or the RTC memory was corrupted (bad CRC), otherwise returns the string written .

wtf · July 30, 2019, 4:10pm

Das hier von manningt (Tom Manning) · GitHub im micropython-repo ebenfalls:

I added the RTC memory implementation to machine_rtc.c, as per the attached file.
The RTC memory is preserved through deepsleep. Not sure if this could be added to the pull.
machine_rtc.c.zip

aus:

wtf · July 30, 2019, 4:41pm

Das LTE-Modem hat nicht-volatilen (NV)-RAM

Möglw. noch viel besser, weil is schon fertig! Der ist für Zertifikate und Private-Keys gedacht, hat jeweils nen Speicherbereich mit Pages dafür. Ich konnt noch keine Aussagen zur Größe finden.

Ist in der AT-Kommando-Referenz vom Sequans-Modem ab Seite 311 (Kapitel 14) beschrieben.

Hab erstmal nur reingeguckt, noch nix geschrieben:

    # LTE: get AT+SQNSNVR="certificate"
    print("[LTE   ] getting AT+SQNSNVR=\"certificate\" ... (LED light green)")
    pycom.rgbled(0x000500)
    try:
      SQNSNVR = lte.send_at_cmd("AT+SQNSNVR=\"certificate\"")
      print("[LTE   ]   ... done: " + SQNSNVR)
    except:
      CGPIAF_at="-999.0"
      print("[LTE   ]   ... failed: " + SQNSNVR)

    # LTE: get AT+SQNSNVR="privatekey"
    print("[LTE   ] getting AT+SQNSNVR=\"privatekey\" ... (LED light green)")
    pycom.rgbled(0x000500)
    try:
      SQNSNVR = lte.send_at_cmd("AT+SQNSNVR=\"privatekey\"")
      print("[LTE   ]   ... done: " + SQNSNVR)
    except:
      CGPIAF_at="-999.0"
      print("[LTE   ]   ... failed: " + SQNSNVR)

Output:

[LTE   ] getting AT+SQNSNVR="certificate" ... (LED light green)
[LTE   ]   ... done: 
+SQNSNVR: "certificate",0,"/CN=root/OU=Service Platform Development department/O=SoftBank Corp./L=Minato-ku/C=JP","000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","/CN=root/OU=Service Platform Development department/O=SoftBank Corp./L=Minato-ku/C=JP","18/02/26 01:33:37","19/02/26 01:33:37","sha256ECDSA","000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","sha1","38db2b4bde4278b9f460cd175ec4c04f88fbc9b8"
+SQNSNVR: "certificate",0,"/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5","000000000000000000000000000000000000000000000000","/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5","06/11/08 00:00:00","36/07/16 23:59:59","sha1RSA","000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","sha1","fdf579b4c78c21872f52909c4282a1a0f4253a95"
+SQNSNVR: "certificate",0,"/C=US/O=VeriSign, Inc./OU=Class 1 Public Primary Certification Authority","00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","/C=US/O=VeriSign, Inc./OU=Class 1 Public Primary Certification Authority","96/01/29 00:00:00","28/08/01 23:59:59","md2RSA","00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","sha1","2142321057ede4c9d432fe974835194c2c52d652"

ERROR

[LTE   ] getting AT+SQNSNVR="privatekey" ... (LED light green)
[LTE   ]   ... done: 
OK

Macht jetz nicht den Eindruck, als ob so nen “Null-Zertifikat” wichtig wär, oder was denken dieGelehrten? Die private-keys sehen jedenfalls sehr leer aus.

Andreas · July 30, 2019, 5:16pm

Ah, im ESP32 Port von Boris Lovosevic (@loboris) ^[1] – exzellent! Vielen Dank für dieses Fundstück.

Siehe auch MicroPython quo vadis. ↩︎

clemens · July 30, 2019, 7:06pm

Auch wenn das LTE-Modem NV-RAM hat sollten wir – falls es für uns taugt – eher das des ESP nehmen, dann funktioniert die Zwischenspeicherung auch mit WiPys, LoPys usw.

clemens · July 30, 2019, 7:50pm

Release Notes:

Added Feautures:

Added RTC memory support

New Features:
[…]

Added support for RTC memory operations Read/Write

roh · July 30, 2019, 8:18pm

ich bin mir nich so sicher… aber ich glaube eher das ist kein eeprom sondern ein flash. bei beidem sind die schreibzyklen beschraenkt. ich faende ja einen kleinen supercap backed ram oder andre nonvolatile speicher mit ohne viel verschleiss besser fuer daten die man >1x am tag schreibt.

Andreas · July 30, 2019, 8:34pm

This got me curious and we found the reason for not finding it at the first hand. While the documentation matching the stable firmware ^[1] version does not tell us anything about the RTC memory, the documentation for the development version ^[2] does.

rtc.memory([data])

Reads RTC memory contents or write data in passed Buffer in to RTC memory.

Example
from machine import RTC
rtc = RTC()
rtc.memory(b'10101010') # writes data in RTC memory
rtc.memory()
Output
b'10101010'

Now, let’s hope it’s really the RTC slow memory, which will hold its contents while the device is in deep sleep. ^[3]

Pycom Firmware & API Reference » RTC module » stable firmware ↩︎
Pycom Firmware & API Reference » RTC module » development firmware ↩︎
ESP32 – Internal Memory
- 8 KBytes SRAM in RTC, which is called RTC SLOW Memory and can be accessed by the co-processor during the Deep-sleep mode.
- 8 KBytes SRAM in RTC, which is called RTC FAST Memory and can be used for data storage; it is accessed by the main CPU during RTC Boot from the Deep-sleep mode.
– Overview of ESP32 features. What do they practically mean? - Tutorials ↩︎

Andreas · July 30, 2019, 11:07pm

Proof

Data stored in RTC slow memory is preserved during deep sleep.

# Before deep sleep
import machine
machine.RTC().memory('Hello World')

# Invoke deep sleep
machine.deepsleep()

# After deep sleep
import machine
machine.RTC().memory()
b'Hello World'

How large is it?

A: The software-addressable RTC slow memory is 2048 bytes in size.

>>> machine.RTC().memory('a' * 2049)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: buffer too long

Notes

The ULP ^[1] also has access to this memory and can use it actively while the main MCU is in deep sleep mode.

PRU und ULP und deren Programmierung ↩︎

clemens · July 31, 2019, 12:34pm

Super, dass es funktioniert und unterstützt wird! Schade, dass es nur 2 kB sind!

Mal ein paar Einschätzungen von unseren beelogger-Kollegen:
https://beelogger.de/?page_id=199005

sie verwenden standardmäßig das 4 kB des EEPROM der RTC, die sie auch verbaut haben
die 1 kB des Atmel 328p sind ihnen auch zu wenig
sie empfehlen ein externes EEPROM oder F-RAM

Bei den beelogger-Datensätzen schreiben sie

erlaubt ein 4kByte EE-Prom mit 5 Minuten Intervall die Speicherung der Daten über 4 Stunden

und

Bei einer täglich einmaligen Datenübertragung mit einem Intervall von 5 Minuten wird ein EE-Prom mit 32kByte (256kbit) benötigt;

Bei meinen open hive-Daten komme ich auf ca. 7 kB Daten pro Tag wenn ich 20minütig messe. 2 kB sind hier ca. 20 Datensätze, allerdings CSV ohne jegliche Metainformation wie Variablenname o.ä.:

2016/08/28 06:21:00,  -0.002, 21.3, 72.5, 21.5, 67.9, 21.0, 21.2, 21.4, 21.4, 21.5, 4.23
2016/08/28 06:41:00,  -0.000, 21.0, 71.6, 21.2, 67.5, 20.8, 21.3, 21.4, 21.4, 21.6, 4.23
2016/08/28 07:01:00,   0.003, 21.0, 70.6, 21.3, 66.5, 21.1, 21.4, 21.5, 21.6, 21.6, 4.23

Wie viele Datensätze können wir bei einem Standard-BOB-System (Waage, 1x BME, 5x DS18B20) speichern ohne großartige andere Daten wie RSSI oder System-Parameter? Vielleicht 10 bis 12 ohne viel Optimierungsarbeit?

Hmmm, ganz subjektiv überlege ich gerade, ob sich der Implementierungs-Aufwand für RTC-Memory lohnt?

wtf · July 31, 2019, 1:55pm

Eine xz-Kompression könnte das um den Faktor 5 schrumpfen lassen.

Das micropython boardmittel uzlib kann wohl nur decompress, in diesem thread hat jemand in C nen pendant gehackt:
https://forum.micropython.org/viewtopic.php?t=5893

Andreas · July 31, 2019, 3:27pm

Wir könnten wohl langsam mal ne inoffizielle Liste von Dingen starten, welche C-Extensions wir beizeiten gerne mit im Programm hätten. Bliebe die Frage, wer sie uns reinhacken kann ;].

wtf · August 1, 2019, 1:00pm

Doesn’t work for me over a powercycle or a machine.reset() (but before/during same run-time). Returns empty (running FiPy v1.20.0.rc11).

wtf · August 1, 2019, 1:23pm

“That memory is only persisted/restored after deep_sleep. It’s erased on reset.” – manningt

Andreas · August 1, 2019, 1:26pm

Yes, this will definitively only work for overcoming the deep sleep as outlined with RTC slow memory deep sleep demo above, while the device will still be powered solidly.

It will not protect against power outage or reset conditions.

Andreas · August 1, 2019, 2:33pm

Flash-based NVRAM for the win

What you probably want to use when talking about the FiPy without external NVRAM is the builtin one, which is actually a partition within the bulk of flash memory.

Using that, you will be able to persistently carry state across reboots or power cycles in all conditions, even after power loss. However, please be aware you should not apply a huge churn on it as everyone knows flash memory will die after a specific amount of read-/write cycles.

So, this memory area should not be used for buffering log data definitively yielding too much churn but might be suitable for humble buffering of measurement data if you know what you are doing.

pycom.nvs_set(key, value)

Set the value of the specified key in the NVRAM memory area of the external flash. Data stored here is preserved across resets and power cycles. Value can only take 32-bit integers at the moment. Example:
import pycom

pycom.nvs_set('temp', 25)
pycom.nvs_set('count', 10)
– Pycom NVRAM API

Weitere Informationen dazu gibt es bei Per MicroPython Zeichenketten ins NVRAM des FiPy speichern.

clemens · January 2, 2020, 2:15am

Why is it 2 kB only while you can read on many pages (e.g.
ESP32 Deep Sleep with Arduino IDE and Wake Up Sources | Random Nerd Tutorials) :

With the ESP32, you can save data on the RTC memories. The ESP32 has 8kB SRAM on the RTC part, called RTC fast memory. The data saved here is not erased during deep sleep.

At least 4 kB should be addressable:

Andreas · March 2, 2020, 2:45pm

github.com

pycom/pycom-micropython-sigfox/blob/01f49f76b7bfe7dcf3d207eb2d0e1b2e90030260/esp32/mods/machrtc.c#L32-L38


/* There is 8K of rtc_slow_memory, but some is used by the system software
    If the USER_MAXLEN is set to high, the following compile error will happen:
        region `rtc_slow_seg' overflowed by N bytes
    The current system software allows almost 4096 to be used.
    To avoid running into issues if the system software uses more, 2048 was picked as a max length
*/
#define MEM_USER_MAXLEN     2048

Increasing MEM_USER_MAXLEN to 6144 or even 4096 indeed does not work. The linker will croak as expected like outlined above.

AR build/FIPY/release/application.a
CPP build/FIPY/release/esp32_out.ld
LINK build/FIPY/release/application.elf
/path/to/xtensa/xtensa-esp32-elf/bin/ld: build/FIPY/release/application.elf section `.rtc.data' will not fit in region `rtc_slow_seg'
/path/to/xtensa/xtensa-esp32-elf/bin/ld: RTC_SLOW segment data does not fit.
collect2: error: ld returned 1 exit status

However, it will compile and link successfully when using 3840 (1024 * 3,75).

#define MEM_USER_MAXLEN     3840

I’ve confirmed this works:

>>> machine.RTC().memory('a' * 3840)