Investigating core panics with BLE on Pycom devices

Coming from BLE auf ESP32 mit MicroPython and BLE auf ESP32 mit MicroPython, @poesel wants to use a GPy device as a data hub / gateway for collecting sensor data from vanilla ESP32 devices using BLE.

On certain circumstances, he is observing core panics (Guru Meditation). We are investigating this here and also reported it at 1.20.0rc13 guru meditation | Pycom user forum.

While we have looked into Random memory corruption faults on ESP32-WROVER rev.1 and rev.2 when running in dual-core mode within the context of Troubleshooting the recent Pycom Firmware Release 1.20.1.r1 @poesel reported these observations even on 1.20.0rc13.


Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Core 0 register dump:
PC      : 0x401da75c  PS      : 0x00060c34  A0      : 0x800921e8  A1      : 0x3ffc0970  
A2      : 0x00000001  A3      : 0x00000001  A4      : 0xb33fffff  A5      : 0x0000cdcd  
A6      : 0x3ffbdafc  A7      : 0x00000000  A8      : 0x8008c97e  A9      : 0x00100000  
A10     : 0x11640000  A11     : 0x00000001  A12     : 0x00000013  A13     : 0x00000000  
A14     : 0x3ffc48fc  A15     : 0x00000002  SAR     : 0x0000000e  EXCCAUSE: 0x00000007  
EXCVADDR: 0x00000000  LBEG    : 0x00000000  LEND    : 0x00000000  LCOUNT  : 0x00000000  
Core 0 was running in ISR context:
EPC1    : 0x40093ee7  EPC2    : 0x00000000  EPC3    : 0x00000000  EPC4    : 0x401da75c

Backtrace: 0x401da75c:0x3ffc0970 0x400921e5:0x3ffc0990 0x400929d7:0x3ffc09b0 0x4009271e:0x3ffc09d0 0x4009304f:0x3ffc09f0 0x4008394d:0x3ffc0a10 0x4008a44d:0x3ffd44a0 0x40081efb:0x3ffd44c0

Even with the latest firmwares taken from Troubleshooting the recent Pycom Firmware Release 1.20.1.r1 [1]

No joy :frowning:

Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (IllegalInstruction). Exception was unhandled.
Memory dump at 0x40201de4: bad00bad bad00bad bad00bad

This is the code which makes my GPy crash reliably: BLE-uPy-Pycom/main.py at master · poesel/BLE-uPy-Pycom · GitHub.


  1. In my case, it was https://packages.hiveeyes.org/hiveeyes/foss/pycom/vanilla/GPy-1.20.1.r1-0.2.0-vanilla-psram-fix.tar.gz. ↩︎

Sorry to hear about that. Then, your code will probably hit a different edge case with BLE. Thanks again for trying our firmware.

Bummer! Further information can be found at [1,2], but I fear that will not help us along.

Personally, I am not able to reproduce the core panic you are observing as I don’t have any BLE devices around advertising theirselves as ‘HEN’ nodes. Sorry that we probably can’t be of any better help here.

[1] Fatal Errors » Panic Handler
[2] Fatal Errors » Guru Meditation Errors

Maybe related:

@mvonflotow and @chegewara reported

1 Like

So, we’ve built yet another series of firmware images from the vanilla upstream sources plus PSRAM-fixes plus CONFIG_FREERTOS_UNICORE and made them available on our package server. The respective build for the GPy can be found at [1].

While this is totally a trial-and-error thing without much substance behind, we will be happy to hear back from you about this, @poesel.


  1. https://packages.hiveeyes.org/hiveeyes/foss/pycom/vanilla/GPy-1.20.1.r1-0.3.0-vanilla-psramfix-unicore.tar.gz ↩︎

Again no joy. :frowning:
@Andreas - even with the discussed modifications. It even went back to crashing after the imports.
Something is really broken here. Hopefully its the software… I’ll have to wait until Espressif and/or Pycom will sort it out.

Strange thing!

Bummer. Sorry to hear that. Good luck with the vanilla ESP32 devices then.

We might want to try decoding the coredump using espcoredump.py as outlined on Guru Meditation Error when connecting to Eduroam (WPA2-Enterprise) | Pycom user forum if you would offer to submit this guy?

./espcoredump.py info_corefile -t b64 -c core_dump.txt firmware.elf

Another report on the Pycom user forum by @shanefowler25.

Even if we are accidentally repeating ourselves here, we scanned the ESP-IDF issues again and want to drop some pointers here.

Both of these proposed solutions look interesting.


wifi + ble crash from time to time (IDFGH-467) · Issue #2724 · espressif/esp-idf · GitHub


frequent Interrupt WDT timeout on core 0. BLE+BT (IDFGH-401) · Issue #2542 · espressif/esp-idf · GitHub

Dear @poesel,

may we humbly ask you to try [1] again. It includes some additional fixes on the ESP-IDF level. If we are lucky, they might actually help here. For more details, enjoy [2].

With kind regards,
Andreas.

[1] https://packages.hiveeyes.org/hiveeyes/foss/pycom/vanilla/GPy-1.20.1.r1-0.5.0-vanilla-butterfly-csfix.tar.gz
[2] Pycom firmware bakery

Sorry, no can do. I didn’t take the GPy with me and I will be at home only end of November.

Thanks for letting us know. You might want to check out [1] when coming back to this.

[1] https://packages.hiveeyes.org/hiveeyes/foss/pycom/vanilla/GPy-1.20.1.r1-0.6.0-vanilla-dragonfly.tar.gz

Dear @poesel,

the Pycom engineers recently published Pycom Firmware Release 1.20.2 (thanks!). I have spotted an update specifically related to BLE.

If we are lucky, this fixes the "bad00bad bad00bad bad00bad" errors you have been observing. So, I am humbly asking you to try GPy-1.20.2.rc3-0.8.0-vanilla-squirrel.tar.gz in order to find out. I will be happy to receive corresponding core dumps.

Thanks already and with kind regards,
Andreas.

Guru Meditation Error: Core  0 panic'ed (Cache disabled but cached memory region accessed)
Guru Meditation Error: Core  0 panic'ed (IllegalInstruction). Exception was unhandled.
Memory dump at 0x4020f450: bad00bad bad00bad bad00bad
...(many pages of the same)...
Guru Meditation Error: Core  0 panic'ed (IllegalInstruction). Exception was unhandled.
Memory dump at 0x4020f450: bad00bad bad00bad bad00bad
Guru Meditation Error: Core  0 panic'ed (IllegalInstruction). Exception was unhandled.
Guru Meditation Error: Core  0 panic'ed (Double exception)
Guru Meditation Error: Core  0 panic'ed (nhandled kernel exception)

Nope, still same :frowning:

Sorry to hear that and thanks again for your feedback on this.

So, it looks like some BLE interaction within BLE-uPy-Pycom/Pycom/main.py at 821d2792dfd8344b17d3757fda6dbae8ba370d67 · poesel/BLE-uPy-Pycom · GitHub is still producing these kinds of fatal errors.