esp-alloc: misreported statistics and *de*alloc failures in PSRAM

[kevinross]

Hi folks,

I've been bruising my forehead on my desk over this for a while now and I'm wondering if I'm doing something wrong here (quite likely, embedded dev is not my day job, my day job uses a substantially higher-level language + platform, but I digress), if there's a bug, if this is expected behaviour, etc.

Using the skeleton project in this gist, I've observed some strange behaviour when trying to allocate memory externally, but only when the external PSRAM is not part of the global heap.

Summary:

• esp-hal rc.0, esp32s3 w/2mb PSRAM
• with feature unified enabled, the stats reported by esp-alloc are accurate regardless of where the vec is stored. Even after filling psram completely, allocations still succeed because internal ram is still available (as expected).
• with feature unified disabled, it appears the memory is never freed, ever. After the vec is dropped, the stats reported are the same as when the vec was live. Any attempt to allocate after psram has reached capacity once fails.

What sent me down this rabbit hole was when I was trying to diagnose an apparent memory leak in my own project's code, where I'm attempting to strip out JSON elements I don't care about while I'm reading from the socket, as opposed to after (payloads can be multiple mb and are not newline delimited standalone JSON docs that every other streaming JSON parser under the sun expects). I kept finding that despite going well out of my way to ensure nothing is carried in memory across awaits that I don't want to be carried, memory usage kept creeping up and up, until eventually the alloc would fail. I also tried using blink_alloc with PSRAM backing it, to ensure the lifetime of the allocations only lasted as long as the local-to-the-function-and-not-global-static proxy allocator did and that I wasn't unintentionally keeping stuff in memory across an await, same result.

This suggests to me that dealloc for the standalone PSRAM allocator I've configured in my test program (and my real project) is a no-op. After the memory is allocated, it can never be freed and reused, and while that is a strategy for optimizing for speed (ie reusing buffers rather than reallocating), it probably shouldn't be built-in behaviour (unless I've completely misunderstood and misread the code & docs & purpose of psram, which isn't impossible).

Thoughts?

[kevinross]

Resolved by this PR