Changes in how memory allocation works since MicroPython v1.11 (to v1.20)

[cole-morrel]

Hello, I am trying to use an OV2640 arducam with my ESP32 and I found this (https://github.com/cj667113/micropython-ov2640) repository that does work. However, it does not work in more recent releases of MicroPython (it gives an error about memory allocation in the ov2640.py file, specifically about lines 79 and 105), and as I've tested it so far, only works in the version it was written in which is 1.11. I would like to update this code so that it works in 1.20, but I'm a little confused what the issue is and how to go about fixing it. The ESP32 I'm using clearly has the necessary memory (as it works on an older version), so I assume this is an issue with how the memory is being managed/allocated. The only other warning I get is about I2C being deprecated, and saying to use SoftI2C, but when I change that it does nothing to fix the issue. Sorry if any of this was confusing, I'm fairly new to MicroPython and still trying to learn. Any help would be greatly appreciated.

[jimmo]

To answer your specific question -- the main thing that has changed between v1.11 and v1.20 with regard to memory is likely to be the total amount of IDF heap available to micropython. In earlier versions of the IDF they laid out things differently in memory which left a larger contiguous region available for microptyhon to use.

That said... this code could be significantly more optimised for memory use. The main thing is that the code uses lists where it should be using bytearray or array.array.

e.g. this line:

picbuf = [ b'\x00' ] * PICBUFSIZE

which allocates a 1950 element list, where each element is a 4-byte integer. Instead it should be

picbuf = bytearray(PICBUFSIZE)

which will be 1/4 the size. Also this should be done once at startup, rather than once per frame, to avoid fragmentation.

Similar sort of stuff in ov2640_constants.py which creates a list of 190 list objects (16 byte each), each with 2 4-byte integers (so 4560 bytes, but should just be a bytes constant

OV2640_JPEG_INIT = b'\xff\x00\x2c\xff...'

taking up 380 bytes instead. The code that accesses this would need to be updated to no longer use the two-level indexing, but that's easy to fix.

There is also a lot of unnecessary small allocations which is going to lead to fragmentation. For example:

def appendbuf(fn, picbuf, howmany):
    try:
        f = open(fn, 'ab')
        c = 1
        #print("writing")
        for by in picbuf:
            if (c > howmany):
                break
            c += 1
            f.write(bytes([by[0]]))
        f.close()
        #print("done writing")
    except OSError:
        print("error writing file")

creates a list and a bytes object for every byte that is written.

Instead, something like:

def appendbuf(fn, picbuf, howmany):
    try:
        f = open(fn, 'ab')
        m = memoryview(picbuf)[0:howmany]
        f.write(m)
        f.close()
    except OSError:
        print("error writing file")

would avoid all of those allocations.

[cole-morrel]

Thank you for the detailed response and explanation, it is very helpful. I'll try to go through and implement the suggestions.

[Wind-stormger]

@cole-morrel Please allow me to consult a loosely related question under this post.

@jimmo Upon seeing your reply, it prompted thoughts about my own code.

from machine import ADC
import math
import asyncio

adc = ADC(4)
adc.width(ADC.WIDTH_12BIT)
adc.atten(ADC.ATTN_11DB)


class TaskManager:
    def __init__(self):

        self.list1 = []
        self.list1_len = 100
        self.average = None

    async def mean(self):
        self.average = sum(self.list1) / self.list1_len

    async def get_list(self):
        while True:
            x = adc.read_uv()
            self.list1.append(x)
            if len(self.list1) > self.list1_len:
                self.list1.pop(0)
            await asyncio.sleep(0)

    async def print_average(self):
        while True:
            if len(self.list1) == self.list1_len:
                await self.mean()
            if self.average is not None:    
                print(self.average)
            await asyncio.sleep(1)


async def main():
    manager = TaskManager()
    task = (manager.get_list(), manager.print_average())
    await asyncio.gather(*task)
    # await asyncio.wait_for(ayncio.gather(*tasks), timeout=30)


asyncio.run(main())

I'm trying to learn asyncio recently and try to apply it, of course this problem may be weakly related to asyncio.

My question is, shouldn't I be using list operations in the coroutine get_list? What might the best code look like?

[jimmo]

@Wind-stormger this isn't particularly related to the advice in the previous question because you're working with integers not bytes.

However this looks like exactly the sort of scenario where you'd be better off using a ring structure (typically called a ring buffer) rather than constantly resizing the list. The idea is you have a fixed size list and two indicies tracking the current head and tail of the list.

Also your mean can be computed iteratively without re-summing the whole list -- just keep the running sum which you add eqch new sample to, and subtract from the sum when you expire an element from the list.

[Wind-stormger]

Thank you for your response. I will start a new thread to discuss the issues I encountered while creating a RingBuffer in MicroPython.

[Wind-stormger]

#12079