Initialising bytearray at specific memory boundary

[czarnikjak]

Is there a way to Initialise a Byte Array at a memory address which is a multiple of its size? ie 1k bytearray starting at 1k memory boundary.
I have a workaround where I initialise an array twice the size of what I need and start writing to it at an offset, but that wastes memory:

myArray = bytearray(2048) #need to write 1024 bytes, but have to allocate 2048
modulo = (uctypes.addressof(myArray)) % 1024
startOffset = (uctypes.addressof(myArray)- modulo + 1024) #element of an array at memory location which is a multiple of 1024

Its OK for 1k, but i will need 32k byterray, that approach would waste a lot of memory.

Any ideas if its possible to do it without allocating bigger array? Or a way to trim the beginning and the end of the array without Python relocating it again?

[dlech]

To do that, I think you are going to have to reserve some memory separate from the MicroPython heap (using the linker script). You could then use the uctypes module or machine.mem8 to access this memory if available.

There isn't a way to tell the existing allocator about what kind of alignment you need.

[czarnikjak]

Thanks, what about something like this:

myArray = uctypes.bytearray_at(0x20014c00,1024)

I don't know if that would play nicely with Micropythons garbage collector and memory management.
In my testing so far it appears that gc.collect() sometimes stamples over this bytearray if initialised like this. Not sure if this is expected behaviour.

[dlech]

Yeah, that is why I said that you need to use the linker script to carve out a memory area that won't be trampled by the heap or stack.

[jimmo]

Yes, MicroPython assumes that all memory not used for .data/.bss is free for it to use for the heap.

As an alternative to using hte linker script, you could just declare a 32k array in .bss (i.e. as a global) and use __attribute__ ((aligned (32768)))

Either way though, to get access to this from Python though you'll need to provide a method to return it as a bytearray -- I'd suggest writing a very short "User C Module".

I'm not aware of a way to do what you want in a nice way without modifications to the firmware though.

BUT

By relying on knowing how the garbage collector works (which is that it searches forward through the heap) you could probably make it work though... Essentially the idea would be to do something like:

# Do this as the very first thing in main.py, before any other imports (i.e. when the heap is essentially empty)
import gc
import uctypes
gc.collect()
a = bytearray()
b = bytearray(0x8000 - 512 - uctypes.addressof(a) % 0x8000)
c = bytearray(0x8000)
del a
del b
gc.collect()
print(uctypes.addressof(c), uctypes.addressof(c) % 0x8000)

The idea is that at startup there will be a small number of allocations quite close to the start of the heap, so the 512 byte allocation ensures that a isn't allocated inside that fragmented region. You probably also want to verify that the second allocation is at least 512 bytes too (which means it must come immediately after a because you now know that a is at the start of the unfragmented "remainder of heap").

This does work on my Pico (and a couple of other stm32 and esp32 boards I tested it on).

(When I tested this and inspected the heap I can see that the b allocation is definitely getting cleaned up in the second gc.collect() but the a allocation is not. I suspect this is because the address of a is still in a CPU register somewhere, but will get cleaned up eventually. I'll investigate in more detail later, but this is not uncommon behavior for the MicroPython GC).

[dlech]

BUT

wow, that terrible and also really clever, lol

[czarnikjak]

Thanks for the detailed response @jimmo, brilliant idea.
It works on my Pico W, but only with a and b sizes of over 1k....i guess this is maybe due to Pico W having IP and BT stack and heap being more fragmented, but anyways it seems to work in my limited testing.

[bidrohini]

You may need to use the uctypes module: https://docs.micropython.org/en/v1.9.3/pyboard/library/uctypes.html