Memory Fault when allocating Thread - Garbage Collection suspected

[mungewell]

I have a project which uses Threads, which one running on each core.

Occasionally I have to restart the thread on the 'other' core... At the time of doing this I occasionally see the following error, may occur after 5 or 6 restarts.

Traceback (most recent call last):
  File "<stdin>", line 676, in <module>
  File "<stdin>", line 440, in OLED_display_thread
  File "libs/umenu.py", line 364, in click
  File "libs/umenu.py", line 333, in select
  File "libs/umenu.py", line 81, in click
  File "libs/umenu.py", line 49, in _call_callable
  File "<stdin>", line 227, in callback_jam
MemoryError: memory allocation failed, allocating 4096 bytes

I am currently running on

RPI_PICO-20240105-v1.22.1.uf2

I suspect that this is due to the garbage collection not being enabled (as per default). This does NOT occur if I force a gc.collect() just before re-starting the thread.
https://github.com/mungewell/pico-timecode/blob/main/main.py#L217

If routinely check the memory usage I see that I am running around 65% used.

Thoughts? Perhaps the Thread code should automatically do garbage collection if the allocation fails and re-try??

[peterhinch]

Allocation failures can occur if the memory becomes fragmented. The attempt to allocate 4096 bytes may have failed because there isn't a free block of contiguous RAM big enough. Options are to pre-allocate and re-use buffers and to periodically issue gc.collect().

[jimmo]

Occasionally I have to restart the thread on the 'other' core

@mungewell What does this mean? How do you trigger the other thread to restart?

I suspect that this is due to the garbage collection not being enabled (as per default).

GC is always enabled on the official builds.

As @peterhinch says, this is likely fragmentation, and the explicit collection allows the collection to happen at a more optimal time (i.e. when there are fewer live objects) which can dramatically help with fragmentation.

You could also experiment with gc.threshold to encourage the GC to run more frequently.

It could also be a bug (there's a lot of complexity around tracing the live objects when threads are involved) so more info would help us investigate further.

Perhaps the Thread code should automatically do garbage collection if the allocation fails and re-try??

This is always the behaviour. Any failed allocation always triggers a collection, and you only see MemoryError if the allocation can still not be satisfied post-collection.

[mungewell]

What does this mean? How do you trigger the other thread to restart?

@jimmo My code is structured (admittedly a little) odd; I have two threads running on the two cores. One thread handles UI, one handles passing data to the PIO blocks, and the bulk of the processing is done in the PIOs.

So the UI will totally shut-down/restart the 'other' thread when user selects option that re-configures the PIOs.
https://github.com/mungewell/pico-timecode/blob/main/main.py#L200

GC is always enabled on the official builds.

nope.
https://docs.micropython.org/en/latest/library/gc.html#gc.threshold

MicroPython v1.22.1 on 2024-01-05; Raspberry Pi Pico with RP2040
Type "help()" for more information.
>>> import gc
>>> gc.threshold()
-1

Like I said, my gc.collect() seems to cure my problem, but reporting as others may be having issues.

[jimmo]

So the UI will totally shut-down/restart the 'other' thread when user selects option that re-configures the PIOs.

Yeah, that's why I'm asking how exactly you achieve this. Are you letting the thread terminate then starting a new thread, or does the thread have a while loop (or equivalent).

nope.

From the docs: "A value of -1 means a disabled allocation threshold."

It does not mean that GC is disabled.

[mungewell]

Yes, the 'other' thread monitors a 'stop' signal and will shut down when signaled. As you can see there is also a sleep loop, to wait for this to happen. Perhaps there needs to be an additional delay for microPython to clean up.

In case you wanted to try, I checked in some more code to make it easier to trigger.

My code will run with/without the Pico-OLED display, but you would need to set up two units - one feeding 'timecode' from GPIO13, to the Unit-Under-Test (UUT) into GPIO18.
https://github.com/mungewell/pico-timecode

Plus apply this patch to make it error, in a short time frame:
error_with_short_calibrate.txt

• When started each unit will blink it's LED at 1PPS, likely these will not be in sync.
• The 'Jam' process is started on the UUT by pressing Key-1/pulling GPIO17 low for 3s. The LED on UUT will stop blinking and the pull-down can be removed.
• After a minute, serial data will start. After a further minute the serial data will stop, signifying that the calibration is complete
• Pressing the button/applying the pull-down will start the "Jam" again, and it is at this point that the error occurs.

If that description didn't make sense I can make a short demo video...

[Josverl]

What is the minimal reproducible example?
Setting up multiple MCUs with connections and a patch makes it harder for others to help you.

[mungewell]

@Josverl You are right, the 'patch' makes it harder... I have uploaded the code (as a branch) with patch attached:
https://github.com/mungewell/pico-timecode/tree/20240309_thread_mem_error

Although I am not sure how much code could be removed whilst still triggering the issue, as I said in earlier posts gc.collect() is a work-around and it's the following _thread.start_new_thread() which gives the error (when run without the collect()).

The code is complex, using 100% of the PIO bocks (which perform most of the work) and helper/UI threads on both cores, I believe that's there something quirking within the '_thread' library.

I uploaded a video to demonstrate the process, and error occurring.
https://youtu.be/uUOWyTvfKwI

[mungewell]

I found another memory issue happening with my code, which I was able to solve.
https://github.com/orgs/micropython/discussions/14143

Marking this Q as solved.