Help nailing down soft OS memory issues on the ESP32-S3 (SD and WiFi don't work consistently, framebuffer hates them.)

[echo-lalia]

TLDR: Launcher program uses WiFi, SDcard, and framebuffer simultaniously, neither SDCard or WiFi work. Without a framebuffer they work. With a quarter sized framebuffer they don't work. Weird janky larger-than-quater-sized framebuffer works, but I hate it. Please help me make quarter framebuffer work <3

---

I'm developing a project for the ESP32-S3 (with no PSRAM, so 512kb RAM total.) and I've been struggling with odd, 'soft' memory issues for a few months now. Some changes I make improve the issues, or make them worse, but I'm struggling to fully understand why, and I would really appreciate some help if anyone is willing to offer it!

Description of the Project:

It's a simple MicroPython app switcher I'm calling MicroHydra.
It's designed for the M5Stack Cardputer (I'd like to expand it for multi-platform support, but that's only tangential to this issue), with some convenient 'OS-like' features builtin.

One major design choice that has been causing particular problems, is that I decided the project should run both as a plain ".py" micropython project, as well as a frozen firmware. Compiling the modules makes all issues go away, so I admit I'm tempted to do away with this self-imposed restriction. However, a lot of users of the software tell me they are using the .py version so they can poke around and see what's going on/makes changes on the fly, which is what's currently stopping me from abandoning that goal.

Description of the issues:

Any time too much memory is used, the WiFi and SDcard modules start misbehaving in inconsistent ways.
Specifically, using a framebuffer, the sound modules, a font module, an SDCard, and WLAN causes the WLAN modules to give OS-Memory errors, or causes either the SDcard, or WLAN, or both, to give some various non-specific OS-errors. These issues specifically pop up in the main launcher.py program, which is designed to sync the clock using WiFi, and find apps on the SDcard.

When I say the issues are "inconsistent" or hard to pin down, I mean it's really hard to tell what will make the issues get better or worse. Sometimes it seems like drastically decreasing the memory usage does not fix the issues, and sometimes increasing memory usage does not make it worse, and I can't tell why.
Restarting the device, or hard power cycling it, can also change the behaviour.

I've rewritten the launcher quite a few times in this process.

I suspect this has something to do with the way the ESP32 port shares memory with ESP-IDF, but beyond that I have no clue what's happening under the surface.

What I've done about

The issues go away entirely if I do not try to use a framebuffer for the display, however the graphics quality suffers quite a bit. (A full framebuffer for the display is 240 * 135 * 2 bytes).
I made an alternate version of the "launcher" app which uses a custom, non-framebuffer version of the display driver, but instead just stored the moving elements in their own framebuffers, and that worked without issue, however it was really janky code that couldn't really be reused, so I wanted to try some other solution.
I completely rewrote the display driver to use a quarter-sized framebuffer (240 * 135 / 2 bytes) which is converted as it's sent to the display. In theory, this framebuffer should be smaller than the partial framebuffers used beforehand, but it doesn't seem to have worked like that, and the WiFi/SDcard issues are back.

Running micropython.mem_info(1) is interesting, but I still struggle to tell exactly what to make of it most of the time. There are a lot of densely packed objects in the memory visualization, but also some spaces. Heap fragmentation may or may not be the culprit; I can't really tell.
Regardless, the memory is never anywhere near 'used up', and I think that's because it's the non-micropython stuff (as in, the esp-idf components) that're running out of space.

What I'm asking for help with

Basically, I'm just asking for any help. If anyone is willing to skim over my code to spot any obvious point where memory issues could be worsened, I'd be super appreciative.
If anyone has additional troubleshooting steps or insight on the topic to offer, I could benefit from that too, I think.
If you have any other form of help to offer, I'll take it.

I would just really like to get the launcher to a point where it's stable with the new display driver, if that's even possible :)

Thanks for reading, and thanks in advance for any help! 😊

[gampam2000]

Hi,

what kind of errors do you get? I maybe have similar errors, I use ans ESP32-S3 Wroom with 8MB Flash and 2MB SPIRAM and get often ESP-IDF errors like:

MPY version : v1.23.0 on 2024-06-08
IDF version : v5.2
Machine : Generic ESP32S3 module with ESP32S3

• Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited). Exception was unhandled.
• Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled.
• and a lot of Resets with:

W (45) boot.esp32s3: PRO CPU has been reset by WDT.
W (50) boot.esp32s3: APP CPU has been reset by WDT.

I also use a lot of RAM aprox. 600-700Kb. I have some rather big arrays with dicts in it and I have seen that sometimes the data gets corrupted.

First i thought it happened because of interrupts but i rewrote everything to async tasks. But the errors persisted. So my guess is also that there is a memory issue and/or ESP-IDF Issue.

My next step is to try to recompile it with different version of the ESP-IDF currently I use v5.2.

[echo-lalia]

Hi! Thanks for the response. Unfortunately though, I think we're having different issues.

The errors I get are usually ENOMEM or something else related to memory, especially when using WiFi.

The errors I'm having have been consistent between MicroPython v1.22 - 1.23, and with the ESP-IDF updates I made when MP.

[glenn20]

ENOMEM exceptions are indeed thrown from the esp wifi stack and can just mean that the preallocated tcp transmit buffers are full. I have on occasion had success with just trapping the exceptions, inserting a short delay and retrying the transmit. However, they may also indicate some unrecoverable error and trapping them wont help.

[peterhinch]

Have you read this doc? It describes well established techniques for minimising RAM fragmentation which I suspect may be at the root of your problem.

[peterhinch]

I've taken a look at your ST7789 driver. You might like to look at this ST7789 driver which also uses a 4-bit LUT to conserve space.

The ST7789 class is subclassed from FrameBuffer enabling the FrameBuffer primitives to be used to draw on the display. Your driver defines multiple tuples: these consume RAM even if declared as const (tuples are immutable whether or not const is used). I appreciate that the aim is readability, but it's often better practice in a RAM-constrained environment to let code comments do the heavy lifting. For a simple example consider:

_ST7789_SWRESET = b"\x01"
# later
        self._write(_ST7789_SWRESET)

While defining lots of glabals is usual practice in device drivers, it is not RAM-efficient. It's more RAM-efficient and (IMO) at least as readable to just write

        self._write(b"\x01")  # Software reset

Writing Python for microcontrollers requires something of a shift in perspective. Typical Python programming practices assume infinite resources and prioritise readability. When coding for microcontrollers you need to constantly have an eye on RAM use (and, in some cases, also performance).

[peterhinch]

After writing the above it occurred to me that my advice re const and tuples might be out of date. MP is constantly being improved. However testing suggests it is a I thought:

1. Declaring a tuple as const has no effect on code size.
2. Precompiling the code containing tuples has no effect (regardless of whether the tuples are const).

I'm listing the test method in case it's of interest to anyone. I ran the following script on a Pyboard 1.1:

import gc
_A = ((
    ( b'\x11', b'\x00', 120),
    ( b'\x13', b'\x00', 0),
    ( b'\xb6', b'\x0a\x82', 0),
    ( b'\x3a', b'\x55', 10),
    ( b'\xb2', b'\x0c\x0c\x00\x33\x33', 0),
    ( b'\xb7', b'\x35', 0),
    ( b'\xbb', b'\x28', 0),
    ( b'\xc0', b'\x0c', 0),
    ( b'\xc2', b'\x01\xff', 0),
    ( b'\xc3', b'\x10', 0),
    ( b'\xc4', b'\x20', 0),
    ( b'\xc6', b'\x0f', 0),
    ( b'\xd0', b'\xa4\xa1', 0),
    ( b'\xe0', b'\xd0\x00\x02\x07\x0a\x28\x32\x44\x42\x06\x0e\x12\x14\x17', 0),
    ( b'\xe1', b'\xd0\x00\x02\x07\x0a\x28\x31\x54\x47\x0e\x1c\x17\x1b\x1e', 0),
    ( b'\x21', b'\x00', 0),
    ( b'\x29', b'\x00', 120)
))
_B = ((
    ( b'\x12', b'\x00', 120),
    ( b'\x13', b'\x00', 0),
    ( b'\xb6', b'\x0a\x82', 0),
    ( b'\x3a', b'\x55', 10),
    ( b'\xb2', b'\x0c\x0c\x00\x33\x33', 0),
    ( b'\xb7', b'\x35', 0),
    ( b'\xbb', b'\x28', 0),
    ( b'\xc0', b'\x0c', 0),
    ( b'\xc2', b'\x01\xff', 0),
    ( b'\xc3', b'\x10', 0),
    ( b'\xc4', b'\x20', 0),
    ( b'\xc6', b'\x0f', 0),
    ( b'\xd0', b'\xa4\xa1', 0),
    ( b'\xe0', b'\xd0\x00\x02\x07\x0a\x28\x32\x44\x42\x06\x0e\x12\x14\x17', 0),
    ( b'\xe1', b'\xd0\x00\x02\x07\x0a\x28\x31\x54\x47\x0e\x1c\x17\x1b\x1e', 0),
    ( b'\x21', b'\x00', 0),
    ( b'\x29', b'\x00', 120)
))

def foo():
    print(_A[0][0] + _B[0][0])  # Force some access
    gc.collect()
    print(gc.mem_free())
foo()

To test I ran mpremote mount .. Between each run I reset the Pyboard, then issued

MicroPython v1.23.0 on 2024-06-02; PYBv1.1 with STM32F405RG
Type "help()" for more information.
>>> import gc
>>> gc.collect()
>>> gc.mem_free()
96352
>>> import rats18
b'\x11\x12'
92736
>>> 96352-92736
3616
>>> 

I repeated with the const declaration removed, also with the script precompiled. All runs used a similar amount of RAM (~3550bytes), with a variation of +-1.8%.

[echo-lalia]

That's great information! Thanks so much for the detailed response.

With your simplified example, am I still correct to assume that:

_ST7789_SWRESET = const(b"\x01")

# later
self._write(_ST7789_SWRESET)

would use the same RAM as:

self._write(b"\x01")  # Software reset

because, the const + leading underscore causes it to be treated as a "real" constant at compile time?

I ask because I'm currently heavily refactoring the project to use magic (auto-filled by a helper script) constants, with the intention of allowing multi-platform support without increasing memory usage. I've been going absolutely wild with constants with the assumption that they would not be taking extra ram after compilation. But, I never actually properly tested that assumption.

[peterhinch]

That is definitely true for int data. I believe it is also true for bytes objects, but I haven't proved this.

[echo-lalia]

Most of the time they're ints anyways, so that's good enough for me :)

Thanks again!

[gampam2000]

Can these ESP-IDF Errors, also be associated with RAM/Heap fragmentation? Or are these all ENOMEM errors?

• Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited). Exception was unhandled.
• Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled.
• and a lot of Resets with:

W (45) boot.esp32s3: PRO CPU has been reset by WDT.
W (50) boot.esp32s3: APP CPU has been reset by WDT.