processing a byte array with constrained memory

[kjm1102]

At first glance

import gc; Q=bytearray(19000)
gc.collect(); b=gc.mem_free(); print(b)
Q[:]=Q[41:]
gc.collect(); a=gc.mem_free(); print(a, 'diff', b-a)
print(len(Q))

168464
168448 diff 16
18959

seems a good way to shrink a bytearray.

But if we constrain the memory

import gc; Q=bytearray(19000)
Z=bytearray(148000)
gc.collect(); b=gc.mem_free(); print(b)
Q[:]=Q[41:]
gc.collect(); a=gc.mem_free(); print(a, 'diff', b-a)
print(len(Q))

20432
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
MemoryError: memory allocation failed, allocating 18959 bytes

it shows that to shrink an array I need free memory a bit bigger than the array.

I can use memoryview to alter bytes in the array without getting an 'allocation failed' but I can't figure out a way to resize with it. Any thoughts on how I might selectively remove bytes from a 20k byte array without needing ~21k of ram to do it?

[GitHubsSilverBullet]

It does not shrink the array. It creates a second one and copies the requested data over.
The new array does get the same ID.
All this stuff is futile, one cannot shrink an array.
Even the: Q[-50000:] = b'' is pretty much useless at the time, because it makes the array smaller but won't give the now utterly useless space back to the memory handler. Also, you won't be able to get the space back unless you destroy/delete the object entirely.

So, at the moment there's only one clear path: Allocate the maximum needed right at the beginning and live with it. Use memoryviews to partition that array if desired.

[rkompass]

MP could benefit from something like a memoryview with a takeover option, that frees the not used parts of an array.
But that wouldn't be compatible with Python of course.

[peterhinch]

The essential feature of all arrays is that the elements occupy contiguous memory locations. This imposes fairly obvious constraints on dynamically resizing them with any efficiency. Consequently Python doesn't allow it. The advice of @GitHubsSilverBullet is correct.

[kjm1102]

I finally managed to shrink this 20k bytearray, without using any extra memory, via assignment slicing. Grateful for all the hand holding, Jimmo in particular. My only gripe is locating the bytes to remove is hard work without .find & .replace, but I eventually worked out how to drag templates over the array to find the bytes that needed to go.

[rkompass]

Would you like to share a code example for this? I googled Python bytearray assignment slicing withouth success. So I'm curious which technique Jimmo showed you :-)

[kjm1102]

https://github.com/orgs/micropython/discussions/14090