Zwischenpufferung von Daten und Log-Informationen im RTC slow memory des ESP32

Die einzigen beiden Referenzen auf RTC_DATA_ATTR im Pycom Universum finden sich bei:

State of the onion with RTC slow memory on ESP32 from Pycom MicroPython

2016

2017

2018

2019

Bottom line

1 Like

Trüffel-schnüffel … das hier riecht interessant:

Scheint ne machine-imlementation in C von nem Individualisten zu sein, aus der Doku:

rtc.write(pos, value)

Write integer (32-bit) value to the position pos in RTC memory.
Return True on success, False if failed.

rtc.read(pos)

Read integer (32-bit) from the position pos in RTC memory.
Returns None if no value has not been written to the RTC integer memory yet or the RTC memory was corrupted (bad CRC), otherwise returns the integer written to the position or 0 (default value).

rtc.write_string(text)

Write the string text to RTC memory.
Return True on success, False if failed.

rtc.read_string()

Read the string from RTC memory.
Returns None if no string has been written to the RTC string memory yet or the RTC memory was corrupted (bad CRC), otherwise returns the string written .

1 Like

Das hier von manningt (Tom Manning) · GitHub im micropython-repo ebenfalls:

I added the RTC memory implementation to machine_rtc.c, as per the attached file.
The RTC memory is preserved through deepsleep. Not sure if this could be added to the pull.
machine_rtc.c.zip

aus:

1 Like

Das LTE-Modem hat nicht-volatilen (NV)-RAM

Möglw. noch viel besser, weil is schon fertig! Der ist für Zertifikate und Private-Keys gedacht, hat jeweils nen Speicherbereich mit Pages dafür. Ich konnt noch keine Aussagen zur Größe finden.

Ist in der AT-Kommando-Referenz vom Sequans-Modem ab Seite 311 (Kapitel 14) beschrieben.

Hab erstmal nur reingeguckt, noch nix geschrieben:

    # LTE: get AT+SQNSNVR="certificate"
    print("[LTE   ] getting AT+SQNSNVR=\"certificate\" ... (LED light green)")
    pycom.rgbled(0x000500)
    try:
      SQNSNVR = lte.send_at_cmd("AT+SQNSNVR=\"certificate\"")
      print("[LTE   ]   ... done: " + SQNSNVR)
    except:
      CGPIAF_at="-999.0"
      print("[LTE   ]   ... failed: " + SQNSNVR)

    # LTE: get AT+SQNSNVR="privatekey"
    print("[LTE   ] getting AT+SQNSNVR=\"privatekey\" ... (LED light green)")
    pycom.rgbled(0x000500)
    try:
      SQNSNVR = lte.send_at_cmd("AT+SQNSNVR=\"privatekey\"")
      print("[LTE   ]   ... done: " + SQNSNVR)
    except:
      CGPIAF_at="-999.0"
      print("[LTE   ]   ... failed: " + SQNSNVR)

Output:

[LTE   ] getting AT+SQNSNVR="certificate" ... (LED light green)
[LTE   ]   ... done: 
+SQNSNVR: "certificate",0,"/CN=root/OU=Service Platform Development department/O=SoftBank Corp./L=Minato-ku/C=JP","000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","/CN=root/OU=Service Platform Development department/O=SoftBank Corp./L=Minato-ku/C=JP","18/02/26 01:33:37","19/02/26 01:33:37","sha256ECDSA","000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","sha1","38db2b4bde4278b9f460cd175ec4c04f88fbc9b8"
+SQNSNVR: "certificate",0,"/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5","000000000000000000000000000000000000000000000000","/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5","06/11/08 00:00:00","36/07/16 23:59:59","sha1RSA","000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","sha1","fdf579b4c78c21872f52909c4282a1a0f4253a95"
+SQNSNVR: "certificate",0,"/C=US/O=VeriSign, Inc./OU=Class 1 Public Primary Certification Authority","00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","/C=US/O=VeriSign, Inc./OU=Class 1 Public Primary Certification Authority","96/01/29 00:00:00","28/08/01 23:59:59","md2RSA","00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","sha1","2142321057ede4c9d432fe974835194c2c52d652"

ERROR

[LTE   ] getting AT+SQNSNVR="privatekey" ... (LED light green)
[LTE   ]   ... done: 
OK

Macht jetz nicht den Eindruck, als ob so nen “Null-Zertifikat” wichtig wär, oder was denken dieGelehrten? Die private-keys sehen jedenfalls sehr leer aus.

Ah, im ESP32 Port von Boris Lovosevic (@loboris) [1] – exzellent! Vielen Dank für dieses Fundstück.


  1. Siehe auch MicroPython quo vadis. ↩︎

Auch wenn das LTE-Modem NV-RAM hat sollten wir – falls es für uns taugt – eher das des ESP nehmen, dann funktioniert die Zwischenspeicherung auch mit WiPys, LoPys usw.

2 Likes

Release Notes:

Added Feautures:

  • Added RTC memory support

New Features:
[…]

  • Added support for RTC memory operations Read/Write
3 Likes

ich bin mir nich so sicher… aber ich glaube eher das ist kein eeprom sondern ein flash. bei beidem sind die schreibzyklen beschraenkt. ich faende ja einen kleinen supercap backed ram oder andre nonvolatile speicher mit ohne viel verschleiss besser fuer daten die man >1x am tag schreibt.

This got me curious and we found the reason for not finding it at the first hand. While the documentation matching the stable firmware [1] version does not tell us anything about the RTC memory, the documentation for the development version [2] does.

Now, let’s hope it’s really the RTC slow memory, which will hold its contents while the device is in deep sleep. [3]


  1. Pycom Firmware & API Reference » RTC module » stable firmware ↩︎

  2. Pycom Firmware & API Reference » RTC module » development firmware ↩︎

  3. ESP32 – Internal Memory

    • 8 KBytes SRAM in RTC, which is called RTC SLOW Memory and can be accessed by the co-processor during the Deep-sleep mode.
    • 8 KBytes SRAM in RTC, which is called RTC FAST Memory and can be used for data storage; it is accessed by the main CPU during RTC Boot from the Deep-sleep mode.

    Overview of ESP32 features. What do they practically mean? - Tutorials ↩︎

1 Like

Proof

Data stored in RTC slow memory is preserved during deep sleep.

# Before deep sleep
import machine
machine.RTC().memory('Hello World')

# Invoke deep sleep
machine.deepsleep()

# After deep sleep
import machine
machine.RTC().memory()
b'Hello World'

How large is it?

A: The software-addressable RTC slow memory is 2048 bytes in size.

>>> machine.RTC().memory('a' * 2049)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: buffer too long

Notes

The ULP [1] also has access to this memory and can use it actively while the main MCU is in deep sleep mode.


  1. PRU und ULP und deren Programmierung ↩︎

2 Likes

Super, dass es funktioniert und unterstützt wird! Schade, dass es nur 2 kB sind!

Mal ein paar Einschätzungen von unseren beelogger-Kollegen:
https://beelogger.de/?page_id=199005

  • sie verwenden standardmäßig das 4 kB des EEPROM der RTC, die sie auch verbaut haben
  • die 1 kB des Atmel 328p sind ihnen auch zu wenig
  • sie empfehlen ein externes EEPROM oder F-RAM

Bei den beelogger-Datensätzen schreiben sie

erlaubt ein 4kByte EE-Prom mit 5 Minuten Intervall die Speicherung der Daten über 4 Stunden

und

Bei einer täglich einmaligen Datenübertragung mit einem Intervall von 5 Minuten wird ein EE-Prom mit 32kByte (256kbit) benötigt;

Bei meinen open hive-Daten komme ich auf ca. 7 kB Daten pro Tag wenn ich 20minütig messe. 2 kB sind hier ca. 20 Datensätze, allerdings CSV ohne jegliche Metainformation wie Variablenname o.ä.:

2016/08/28 06:21:00,  -0.002, 21.3, 72.5, 21.5, 67.9, 21.0, 21.2, 21.4, 21.4, 21.5, 4.23
2016/08/28 06:41:00,  -0.000, 21.0, 71.6, 21.2, 67.5, 20.8, 21.3, 21.4, 21.4, 21.6, 4.23
2016/08/28 07:01:00,   0.003, 21.0, 70.6, 21.3, 66.5, 21.1, 21.4, 21.5, 21.6, 21.6, 4.23

Wie viele Datensätze können wir bei einem Standard-BOB-System (Waage, 1x BME, 5x DS18B20) speichern ohne großartige andere Daten wie RSSI oder System-Parameter? Vielleicht 10 bis 12 ohne viel Optimierungsarbeit?

Hmmm, ganz subjektiv überlege ich gerade, ob sich der Implementierungs-Aufwand für RTC-Memory lohnt?

Eine xz-Kompression könnte das um den Faktor 5 schrumpfen lassen.

Das micropython boardmittel uzlib kann wohl nur decompress, in diesem thread hat jemand in C nen pendant gehackt:
https://forum.micropython.org/viewtopic.php?t=5893

1 Like

Wir könnten wohl langsam mal ne inoffizielle Liste von Dingen starten, welche C-Extensions wir beizeiten gerne mit im Programm hätten. Bliebe die Frage, wer sie uns reinhacken kann ;].

Doesn’t work for me over a powercycle or a machine.reset() (but before/during same run-time). Returns empty (running FiPy v1.20.0.rc11).

“That memory is only persisted/restored after deep_sleep. It’s erased on reset.” – manningt

Yes, this will definitively only work for overcoming the deep sleep as outlined with RTC slow memory deep sleep demo above, while the device will still be powered solidly.

It will not protect against power outage or reset conditions.

Flash-based NVRAM for the win

What you probably want to use when talking about the FiPy without external NVRAM is the builtin one, which is actually a partition within the bulk of flash memory.

Using that, you will be able to persistently carry state across reboots or power cycles in all conditions, even after power loss. However, please be aware you should not apply a huge churn on it as everyone knows flash memory will die after a specific amount of read-/write cycles.

So, this memory area should not be used for buffering log data definitively yielding too much churn but might be suitable for humble buffering of measurement data if you know what you are doing.

Weitere Informationen dazu gibt es bei Per MicroPython Zeichenketten ins NVRAM des FiPy speichern.

2 Likes

Why is it 2 kB only while you can read on many pages (e.g.
ESP32 Deep Sleep with Arduino IDE and Wake Up Sources | Random Nerd Tutorials) :

With the ESP32, you can save data on the RTC memories. The ESP32 has 8kB SRAM on the RTC part, called RTC fast memory. The data saved here is not erased during deep sleep.

At least 4 kB should be addressable:

Increasing MEM_USER_MAXLEN to 6144 or even 4096 indeed does not work. The linker will croak as expected like outlined above.

AR build/FIPY/release/application.a
CPP build/FIPY/release/esp32_out.ld
LINK build/FIPY/release/application.elf
/path/to/xtensa/xtensa-esp32-elf/bin/ld: build/FIPY/release/application.elf section `.rtc.data' will not fit in region `rtc_slow_seg'
/path/to/xtensa/xtensa-esp32-elf/bin/ld: RTC_SLOW segment data does not fit.
collect2: error: ld returned 1 exit status

However, it will compile and link successfully when using 3840 (1024 * 3,75).

#define MEM_USER_MAXLEN     3840

I’ve confirmed this works:

>>> machine.RTC().memory('a' * 3840)
1 Like