Timing things on MicroPython for ESP32

Entwicklung / Development Firmwareschmiede / Firmware development
1
Breadcrumb

We are coming from Untersuchung und Verbesserung des Timings bei der Ansteuerung der DS18B20 here.

Einleitung

Auch wenn wir schon seit längerem evangelisieren, wie mit den Ressourcen an der Grenze zwischen Hardware und Software am exzellentesten umgegangen werden sollte

haben wir es in der Praxis weiterhin des öfteren mit Problemen beim Timing zu tun. Der Teufel steckt im Detail und viele der entsprechenden Aspekte wurden in der Adventszeit des Arduino-Universums nicht ordentlich berücksichtigt und führen sowohl zu konkreten Altlasten bei der Umstellung auf neue Prozessoren und Betriebssysteme als auch zu mentalen Anfälligkeiten im Sinne von “einfach weiter wie bisher”.

Und nun?

Über die Unmöglichkeit, ordentliches Timing auf unmodifizierten SBC-Systemen mit regulären Linux-Kerneln (RaspberryPi, BeagleBone) hinzukriegen, wollen wir nicht weiter eingehen, sondern sind froh, dass mittlerweile einige der ursprünglich rein für 8-bit AVR-Prozessoren o.ä. ausgelegten Bibliotheken erwachsener geworden sind und mittlerweile auch auf 32-bit Systemen mit höherer Interrupt-Last aufgrund von eingebetteten Peripheriesubsystemen ebenfalls gut ihren Dienst tun. [1]

Während es bisher so war, dass der Programmcode direkt auf dem Eisen lief, hat man heutzutage auf größeren Maschinen meist noch ein Betriebssystem zwischen Anwendung und Hardware. Darauf muss bei der Programmierung Rücksicht genommen werden.

Solange noch in systemnahen Sprachen wie C, C++ oder Rust entwickelt wird, reicht es normalerweise, sich mit den Umständen des Betriebssystems und dem Datenblatt des digitalen Sensors zu beschäftigen und bei der Implementierung die Interruptbehandlung per Protect code when accessing shared resources ordentlich zu machen.

Hier soll nun die Reise weitergehen zu Problemaspekten, die man sich bei der Programmierung von a) Multicore-Maschinen mit Sprachen/Programmierumgebungen einhandelt, die nun b) auch noch einen Garbage-Kollektor unter der Haube haben, der noch viel ungünstiger auf das Timing Einfluß nehmen kann als ein paar querschlagende unmaskierte Interrupts. Meow.

  1. An dieser Stelle wollen wir kurz die von Rob Tillart überarbeitete DHTNEW-Bibliothek nennen, über die wir bei DHT22 hängt immer nach Arduino Firmware Reset berichteten sowie den ebenfalls überarbeiteten Klassiker GitHub - bogde/HX711: An Arduino library to interface the Avia Semiconductor HX711 24-Bit Analog-to-Digital Converter (ADC) for Weight Scales., an dem wir jüngst selbst ein wenig mit angepackt haben.

    Unabhängig davon soll hier nicht unerwähnt bleiben, dass es für beide genannte Sensoren solidere Alternativen gibt, die bevorzugt verwendet werden sollten. Wir haben bei uns BME280 sowie ADS1231 dafür erschlossen. ↩︎

1 Like
2

Beobachtungen durchs Oszilloskop

Wir wollen hier noch einmal wesentliche Forschungsarbeiten von @robert-hh aus dem Pycom Forum wiedergeben. Dankenswerterweise ist er der Angelegenheit bereits mit einem Oszi zu Leibe gerückt und hat damit @weef und @roh Arbeit erspart. Thanks a bunch.

1. Garbage collector makes us burst in tears

Introduction

With his first tests published in spring 2018, he observes a test program producing a 20µs pulse and then consuming some memory, repeating both within a tight loop. This process is heavy on both cpu resources as well as memory allocation and deallocation, so any glitches coming from not having a solid foundation scheduling things appropriately will be observable right away. We will see how MicroPython’s garbage collector will well interrupt your program and stop its world intermittently, in turn mostly producing erroneous timings without further ado.

We will see that manually invoking the garbage collector by calling gc.collect() right before the initial pulse improves the situation significantly. This is a countermeasure to somehow control the garbage collector and prevent it from interrupting your program at arbitrary times, which is just bad when doing things which require appropriate timing.

However, please recognize this is just a workaround. The best thing in this environment is to not having any memo

Observation / Problem

These results are not good. There is no pulse shorter than ~40µs, sometimes extending to over 500µs on the FiPy.

image
image760×570 129 KB

Garbage collector control

The longest pulse seen for the FiPy is now 65µs. Please note that the time scale has changed in this picture.

image
image760×570 152 KB

2. Tuning timings in the onewire.py library

Introduction

Observation / Problem / Mitigation

The sleep_us(1) calls in Pycom’s onewire.py occasionally extend the wait period to about 30µs, effectively missing the read window at 15µs (left image).

Without one invocation of that sleep, the duration of the initial pulse is sometimes still extended, but much more seldom and shorter than 10µs, being still in the safe range (right image).

image
image760×570 164 KB
image
image760×570 153 KB

Considerations

1 Like
3

Still reading? You know who you are.

While observations like outlined above might make designers and users of PLC / SPS systems both laugh out loud and burst in tears at the same time, these are actually obvious things which should always be anticipated, especially in this very environment.

Actually, who would even think of doing sampling and toggling directly in (Micro-)Python at all? No matter what, in this spirit you will not be surprised to hear that

.
.
.
.
.
. .
. . .
|
|
|
||
|||

image
This is obviously a squirrel. – Wat. Destroy all software.

4

;)
qed.

Is there something like a disable_irq() for the garbage collector?

At least temporary or with a timeout for i2c or onewire, a few milliseconds of uninterrupted CPU time should be enough as long as you do not exhaust memory of course.

But that’s the same rules as in C:

  • do not do any dynamic memory allocations in tight loops and irq handlers.
  • use const size objects or objects allocated outside of the scope of the code protected by disable_irq()/enable_irq()
  • get done fast and exit in deterministic time.
  • avoid anything using strings and especially formatstrings

I hope something similar could work as well for us in MicroPython. :)

1 Like
6

A post was merged into an existing topic: Untersuchung und Verbesserung des Timings bei der Ansteuerung der DS18B20 Sensoren unter MicroPython

7

time.sleep_us()?

8
9

Control of Garbage Collection

A GC can be demanded at any time by issuing gc.collect() . It is advantageous to do this at intervals, firstly to pre-empt fragmentation and secondly for performance. A GC can take several milliseconds but is quicker when there is little work to do (about 1ms on the Pyboard). An explicit call can minimise that delay while ensuring it occurs at points in the program when it is acceptable.

Automatic GC is provoked under the following circumstances. When an attempt at allocation fails, a GC is performed and the allocation re-tried. Only if this fails is an exception raised. Secondly an automatic GC will be triggered if the amount of free RAM falls below a threshold. This threshold can be adapted as execution progresses:

gc.collect()
gc.threshold(gc.mem_free() // 4 + gc.mem_alloc())

This will provoke a GC when more than 25% of the currently free heap becomes occupied.

In general modules should instantiate data objects at runtime using constructors or other initialisation functions. The reason is that if this occurs on initialisation the compiler may be starved of RAM when subsequent modules are imported. If modules do instantiate data on import then gc.collect() issued after the import will ameliorate the problem.

– MicroPython on microcontrollers — MicroPython latest documentation

Heap defragmentation

When a running program instantiates an object the necessary RAM is allocated from a fixed size pool known as the heap. When the object goes out of scope (in other words becomes inaccessible to code) the redundant object is known as “garbage”. A process known as “garbage collection” (GC) reclaims that memory, returning it to the free heap. This process runs automatically, however it can be invoked directly by issuing gc.collect() .

The discourse on this is somewhat involved. For a “quick fix”, issue the following periodically:

gc.collect()
gc.threshold(gc.mem_free() // 4 + gc.mem_alloc())

– https://docs.micropython.org/en/latest/reference/constrained.html#the-heap

Accessing hardware directly

Note:
Code examples in this section are given for the Pyboard. The techniques described however may be applied to other MicroPython ports too.

This comes into the category of more advanced programming and involves some knowledge of the target MCU. Consider the example of toggling an output pin on the Pyboard. The standard approach would be to write

mypin.value(mypin.value() ^ 1) # mypin was instantiated as an output pin

This involves the overhead of two calls to the Pin instance’s value() method. This overhead can be eliminated by performing a read/write to the relevant bit of the chip’s GPIO port output data register (odr). To facilitate this the stm module provides a set of constants providing the addresses of the relevant registers. A fast toggle of pin P4 (CPU pin A14 ) - corresponding to the green LED - can be performed as follows:

import machine
import stm
BIT14 = const(1 << 14)
machine.mem16[stm.GPIOA + stm.GPIO_ODR] ^= BIT14

– http://docs.micropython.org/en/v1.9.3/pyboard/reference/speed_python.html#accessing-hardware-directly

10

Looks like Pycom MicroPython allows disabling the garbage collector already.

gc.disable()

Disable automatic garbage collection. Heap memory can still be allocated, and garbage collection can still be initiated manually using gc.collect() .

– gc

11

nice… so whats the plan then?

gc.disable() 
gc.collect()
<do timing critical stuff>
gc.enable()

?

12

¡Sí, claro!

1 Like
13

wolltest Du erstmal weglassen?

14

Das passiert an anderer Stelle im Code, wurde aber vorhin ebenfalls verbessert.

15

Jacob Beningo wrote about his experiences re. realtime capabilities with MicroPython on a STM32 around 2015 already.

17

If you are still reading here, you might enjoy the discussion about interrupt handling accurateness with MicroPython and related things to consider.

All people interested in this topic will probably enjoy reading the canonical documentation about Writing interrupt handlers with MicroPython.

18

Just another Fundstück.

See also ESP32 with Arduino Framework Freezes · Issue #11 · siara-cc/esp32_arduino_sqlite3_lib · GitHub.

19

RMT is coming to Genuine MicroPython.

– via: CRC err DS18B20 OneWire ESP32 - MicroPython Forum