Memory use when accessing array containing > 30-bit values

[pdw-mb]

I spent quite a long time pinning down why the Neopixel driver I was using was occasionally garbage collecting mid-update, causing LED glitches.

This shows the underlying cause, which is that accessing array members with values of 2**31 or more results in a heap allocation:

import gc
import array

def test(arr):
    before = gc.mem_free()
    print(arr[0])
    print(gc.mem_free()-before)

arr = array.array('I', [0])
arr[0] = 0x3FFFFFFF
test(arr)
arr[0] = 0x40000000
test(arr)

Produces (on the rp2 port, at least):

1073741823
0
1073741824
-32

I've now found a better way around the original problem (see #10465), but I'm curious to know what's going on here, and if there are better ways to work with full 32-bit numbers.

[dpgeorge]

On 32-bit ports that use object representation A (most ports), integers between -230 and 230-1 are "small integers" and are stored directly in the object handle, ie they don't need any heap memory. Integers outside this range are "big integers" and require heap memory (and more memory the larger the value).

Ways around this:

1. Rebuild the firmware with object representation D. Then floats and integers up to 2**46 do not require heap memory. Downside is that everything runs a bit slower, and everything else uses about 1.5x memory.
2. Use array('H') and 16-bit numbers only.
3. Do all the logic that needs 32-bit numbers with @micropython.viper functions.

[peterhinch]

I've just twigged: the message from Damien had a formatting bug. This fragment should read

integers between -2**30 and 2**30-1 are "small integers"

[robert-hh]

@peterhinch The formatting glitch was obvious.
@pdw-mb Yes, the extra memory holing the value of 0x40000000 is just used for the temporary print() argument. The array itself continues to hold 32 bit integers. If in a call to the neopixel driver you get glitches when calling gc.collect(), then it may be that the array object went out of scope before writing of the driver to the NeoPixel devices was finished.

[pdw-mb]

The issue I was having was that GC was occasionally happening during the loop that writes to the PIO TX FIFO, which was:

for n in range(pixels):
   sm.put(pixels[n])

The GC caused a delay long enough for the LEDs to assume a reset. Once I pinned down that it was GC that was triggering the problem, I was able to initially fix it with an explicit gc.collect() before starting the loop. I also tried using your handy PIO DMA helper functions (which worked well - thank you! - although I struggled to get the PIO code working correctly for 24-bit LEDs) before discovering that I can actually just pass the array straight to sm.put() and avoid the problem completely.

As bugs go, it was entertainingly subtle, as it only affected four colour LEDs (RGBW) where the first colour was at more than 25% brightness, and where LEDs later in the string were being set to a different colour :)

It'd be helpful to add something to this section on this issue, as I spent time staring at that trying to figure out why this heap allocation was happening. I'm happy to have a go at drafting something based on the above.

[pdw-mb]

Thanks for the reply @dpgeorge . I've attempted to capture your answer in #10486, which I think would have helped me when I was grappling with the bug.

[jimmo]

I often wonder if perhaps the solution for this is to make framebuf support an RGB and RGBW format instead of using array.array.