Create and initialise a bytearray

[obdevel]

Creating an initialised bytearray in what I understand to be a pythonic way uses significantly more memory. e.g.

e = bytearray([255] * (255 * 24))

Consumes around 30KB according to micropython.mem_info()

The 'anti-pattern':

e = bytearray(255 * 24)
for i in range(len(e)):
  e[i] = 255

Consumes around 6KB.

Both appear to have the same result. What am I doing wrong here ?

[peterhinch]

This code fragment [255] * (255 * 24) produces a list instance with 6120 elements, each of which is a 4 byte quantity (a small int). So this uses 24,480 bytes. Add the 6120 bytes for the bytearray and you get to 30,600 bytes plus some overhead. Issuing

e = bytearray(0xFF for _ in range(255*24))

is the economical approach. This creates an iterator which is much more efficient.

[obdevel]

Thanks Peter. That'll teach me to use stack overflow, regardless of the score that answer received ! I suppose what is considered pythonic when you have GBs of RAM is different to when you don't.

[mcauser]

So, there's pythonic and micropythonic, got it.

[peterhinch]

It's always worth checking the date on stack overflow posts. The iterator syntax is a relatively new -and excellent - addition to CPython.

[cwalther]

Is this case optimized internally so that (Micro-)Python knows the length of the generator and can allocate the bytearray in one go? Because for a generic generator, it certainly can’t know the length and, I guess, needs to reallocate the bytearray again and again as it iterates and gets more and more elements.

[rkompass]

It seems so, from my tests at least (had the same question in the past).

[jimmo]

@cwalther @rkompass MicroPython doesn't know the length of the generator in this case, and so this will realloc the underlying buffer (many times!). However the realloc will always attempt to grow into adjacent unused heap without moving, so in many cases this will essentially be no more expensive than if it had done the full allocation. But in a case where the heap is a bit fragmented, this will be expensive.

(MicroPython does have an optimisation here where if the generator does know its length then it does pre-allocate. For example bytearray(range(N)) would trigger this. But that doesn't help you here.