Guidance for memory management in a UserMod

[ma261065]

Hi all,

I'm porting some existing C code as a User Module, and am unclear how to best translate its memory management functions.

At the top of the existing code it has the following defines:

#define __malloc_heap_psram(size) heap_caps_malloc_prefer(size, 2, MALLOC_CAP_DEFAULT | MALLOC_CAP_SPIRAM, MALLOC_CAP_DEFAULT | MALLOC_CAP_INTERNAL)

#define __calloc_heap_psram(ch, size) heap_caps_calloc_prefer(ch, size, 2, MALLOC_CAP_DEFAULT | MALLOC_CAP_SPIRAM, MALLOC_CAP_DEFAULT | MALLOC_CAP_INTERNAL)

and throughout the code there are calls to malloc(...), __malloc_heap_psram(...) and __calloc_heap_psram(...). The code logic seems to be that if they need a few bytes here and there they will use malloc(...), but for larger buffers they use __malloc_heap_psram(...)

From what I have read, it is unsafe to use the standard malloc(...) functions, as MicroPython doesn't have visibility of the memory having been allocated and will likely stomp on it at some point. So I plan to replace the malloc(...) calls with with mp_obj_malloc(...)

However I'm less clear on the calls to __malloc_heap_psram(...) and __calloc_heap_psram(...). I think they are using the underlying ESP-IDF to do malloc(...), but ensuring that the allocations happen in PSRAM. Can I also presume that MicroPython will stomp on those eventually? Is it safe to also replace them with calls to mp_obj_malloc(...)?

My logic is that MicroPython "knows about" PSRAM so it will allocate any larger buffers there anyway. Or am I being naive in my understanding?

Finally, the __calloc_heap_psram(...) calls... I presume there is no equivalent mp_obj_calloc(...)? So I would have to write a helper function in C to do a mp_obj_malloc(...) and then initialise the returned memory? Or is there a smarter way of doing that?

[karfas]

On the ESP32, Micropython, as well as most other ESP-IDF/freertos task, allocates it's memory from freertos, so I don't see any problem to use malloc() or one of the other allocation functions. Nothing from micropython will "stomp" on the so allocated memory (assuming there are no bugs causing this and no micropython functions like mem32() are used to patch arbitrary memory locations).

mp_obj_malloc() will allocate from the micropython heap. There might be additional steps required to protect this memory from the garbage collector.

See ports/esp32/main.c and py/gc.c for details.

[jimmo]

MicroPython will not stomp on your IDF heap allocations -- the issue is more that the MicroPython GC cannot trace this memory, so if you have pointers here that point into MicroPython-heap allocations, they will not be kept alive.

So allocating on the IDF heap for something like a framebuffer is totally fine.