Memory consumption with array('B'...

[mendenmh]

in MicroPython '3.4.0; MicroPython v1.20.0 on 2023-04-26', on an RPi Pico,

I would assume that an object such as array('B', range(x)) would take up 'x' bytes plus the header. It seems to take massively more, as seen below. My 65536 element array makes about 98k bytes disappear from the heap. It all reappears properly when I delete the object and re-collect, so it isn't a leak.

I note that bytearray() does consume the correct amount, while array.array('B'...) does not. For practical purposes, I probable would use bytearray anyways, but what is happening with array.array?

Does anyone know why this is so, or why it must be (if it isn't a bug)?

Thanks much.

>>> gc.collect()
>>> micropython.mem_info()
stack: 524 out of 7936
GC: total: 226560, used: 17424, free: 209136
 No. of 1-blocks: 208, 2-blocks: 39, max blk sz: 74, max free sz: 6424
>>> bb = array('B', range(65536))
>>> bbarr = uctypes.addressof(bb)
>>> f"{bbarr:08x}"
'20026e80'
>>> gc.collect()
>>> micropython.mem_info()
stack: 524 out of 7936
GC: total: 226560, used: 115952, free: 110608
 No. of 1-blocks: 162, 2-blocks: 36, max blk sz: 4096, max free sz: 1424
>>> del bb
>>> gc.collect()
>>> micropython.mem_info()
stack: 524 out of 7936
GC: total: 226560, used: 15936, free: 210624
 No. of 1-blocks: 204, 2-blocks: 39, max blk sz: 74, max free sz: 6424
>>> bb = array('B', range(65536*2))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError: memory allocation failed, allocating 131072 bytes

[shariltumin]

I did not experience the problem you mentioned. Here are the results of my test:

>>> 
MPY: soft reboot
MicroPython v1.20.0 on 2023-04-26; Raspberry Pi Pico W with RP2040
Type "help()" for more information.
>>>
>>> import gc
>>> from array import array
>>> import micropython as mp
>>> gc.collect()
>>> m0 = gc.mem_free()
>>> m0
188112
>>> bb = array('B', range(180000))
>>> m1 = gc.mem_free()
>>> m0-m1
180608
>>> m1
7504
>>> mp.mem_info()
stack: 540 out of 7936
GC: total: 192768, used: 185744, free: 7024
 No. of 1-blocks: 84, 2-blocks: 19, max blk sz: 11250, max free sz: 425
>>> gc.collect()
>>> gc.mem_free()
7968
>>> mp.mem_info()
stack: 540 out of 7936
GC: total: 192768, used: 184960, free: 7808
 No. of 1-blocks: 54, 2-blocks: 14, max blk sz: 11250, max free sz: 431

The only explanation I can think of for your case is that the memory used for range(65536) has not yet been collected by gc when you run micropython.mem_info().

[mendenm]

@shariltumin OK, something is definitely squirrelly. See two cases below. My original test case, with 65536 bytes, eats almost as much memory as your test case with 180,000!

I was careful here to do plenty of gc.collect() to make sure it wasn't in some recoverable block.

original case, 65536 bytes allocated, insane memory usage:

MicroPython v1.20.0-489-ga3862e726 on 2023-09-20; Raspberry Pi Pico with RP2040
Type "help()" for more information.
>>> 
>>> from micropython import mem_info
>>> import gc
>>> from array import array
>>> mem_info()
stack: 500 out of 7936
GC: total: 225472, used: 17040, free: 208432
 No. of 1-blocks: 359, 2-blocks: 65, max blk sz: 64, max free sz: 13012
>>> gc.mem_free()
206912
>>> a=array('B', range(65536))
>>> gc.mem_free()
15280
>>> gc.collect()
>>> gc.mem_free()
12944
>>> mem_info()
stack: 500 out of 7936
GC: total: 225472, used: 212528, free: 12944
 No. of 1-blocks: 134, 2-blocks: 23, max blk sz: 8655, max free sz: 412
>>> del a
>>> mem_info()
stack: 500 out of 7936
GC: total: 225472, used: 215104, free: 10368
 No. of 1-blocks: 207, 2-blocks: 30, max blk sz: 8655, max free sz: 412
>>> gc.mem_free()
7760
>>> gc.collect()
>>> gc.mem_free()
215856
>>> gc.collect()
>>> mem_info()
stack: 500 out of 7936
GC: total: 225472, used: 9632, free: 215840
 No. of 1-blocks: 168, 2-blocks: 26, max blk sz: 64, max free sz: 12948
>>> 

case with 180k bytes allocated, looks fairly normal.

>>> 
────────
MicroPython v1.20.0-489-ga3862e726 on 2023-09-20; Raspberry Pi Pico with RP2040
Type "help()" for more information.
>>> 
>>> from micropython import mem_info
>>> import gc
>>> from array import array
>>> gc.mem_free()
207312
>>> a=array('B', range(180000))
>>> gc.mem_free()
14880
>>> gc.collect()
>>> gc.mem_free()
12560
>>> 
>>> del a
>>> gc.collect()
>>> gc.mem_free()
215856
>>> 

[mendenm]

Just to continue the previous session without any restart, (so the gc.mem_free -> 215856 is still valid)

>>> a=array('B', range(65536))
>>> gc.collect()
>>> gc.mem_free()
9152
>>> del a
>>> gc.collect()
>>> gc.mem_free()
215584
>>> a=array('B', range(65537))
>>> gc.collect()
>>> gc.mem_free()
9120
>>> del a
>>> gc.collect()
>>> gc.mem_free()
215520
>>> a=array('B', range(180000))
>>> gc.collect()
>>> gc.mem_free()
9136
>>> del a
>>> gc.collect()
>>> gc.mem_free()
215520
>>> 

which very directly shows the problem. I was wondering if the original 65536 issue was state-dependent, and would behave right after doing the big allocation. It is not the case. The missing memory on 65536 (and, just to see if there was something especially funny going on with 65536 allocations, 65537) sized arrays is entirely reversible. I will poke a little harder at this, because at this point I think there really is a bug either in the allocator or the reporter.

[mendenm]

so, I restarted micropython and did this:

MicroPython v1.20.0-489-ga3862e726 on 2023-09-20; Raspberry Pi Pico with RP2040
Type "help()" for more information.
>>> import gc
>>> from array import array
>>> gc.mem_free()
209936
>>> a=array('B', range(65536))
>>> gc.collect()
>>> gc.mem_free()
9904
>>> b=array('B', range(65536))
>>> gc.collect()
>>> gc.mem_free()
9648
>>> c=array('B', range(65536))
>>> gc.collect()
>>> gc.mem_free()
9408
>>> d=array('B', range(65536))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError: memory allocation failed, allocating 65536 bytes
>>> del a,b,c
>>> gc.collect()
>>> gc.mem_free()
215904
>>> 

It appears that gc.mem_free is returning entirely bogus numbers, even after gc.collect. With only 9k free after one allocation, I succeeded in allocating two more 65536 byte arrays. Trying to allocate a fourth failed, as expected.

I have elevated this to a bug report.

[jimmo]

See #12557 (comment)