MemoryError on Pico eventhough free memory is sufficient

[ondrej1024]

Hi,
Sometimes I get a MemoryError exception on the Pico, saying that it cannot allocate N bytes. But when checking the available memory with gc.mem_free() it reports more free memory than what is needed. Typically this happens when using the networking functions from the usocket package.

Someone has an explanation for this?

Thanks.

[peterhinch]

This can occur due to RAM fragmentation. If the application requires N contiguous bytes it will fail if the largest block of free space is smaller than that. In some cases calling gc.collect() periodically can reduce fragmentation. Attention to detail in coding can help greatly: create any large buffers early and re-use them; i.e. don't keep re-creating object if they can be re-used.

[ondrej1024]

Thanks for your reply. This sounds reasonable. In fact I am suspecting my web server implementation to be the root cause of the fragmentation. It receives a buffer of data each time an HTTP request is made from a client and processes it.

Here is a code snippet which receives the request from a socket:

def recv_http_request(c):
    MAX_PACKET_LEN = 800  # 800 seems to be the max chunk size
    data = bytearray()
    # Receive request in chunks
    while True:
        packet = c.recv(MAX_PACKET_LEN)
        if not packet:
            break
        data.extend(packet)
        print("webserver: received chunk of %d bytes" % (len(packet)))
        if len(packet) < MAX_PACKET_LEN:
            break
    # Get method and url
    data = data.decode().split(' ',2) 
    return (data[0],data[1])

I am not sure if the data bytearray and packet variable are handled correctly. Could this code cause the RAM fragmentation?

[peterhinch]

It should be possible to re-use a buffer. Instantiate a buffer big enough to accept the longest response. Declare the buffer early in execution, before fragmentation has set in. You also need a memoryview pointed at the buffer.

You can use s.readinto() to populate the buffer. The memoryview enables you to take allocation-free slices of the buffer. Say you have received a total of N bytes then mv[:N] provides the slice containing the response.

[ma261065]

"This will create three memoryview objects and no copies of the data."

For clarity, in 2, I was trying to show that the mv object is created once, at the same time as the buffer. So the data1/data2 lines would create two more mv objects.

However, the statement that no copies of the data are made for 2) - the memoryview doco says that:

Further, the slice syntax a:b causes further allocation by instantiating a slice(a, b) object

Isn't that in effect a copy of the data?

[jimmo]

So the data1/data2 lines would create two more mv objects.

That's right. But memoryview objects are small (16 bytes).

Further, the slice syntax a:b causes further allocation by instantiating a slice(a, b) object
Isn't that in effect a copy of the data?

No, a slice in Python is an object that represents the start/end/step values passed to the subscript operator.

So when you write

data1 = mv[:N]

this is actually

s = slice(None, N)
data1 = mv.__getitem__(s) # returns a new memoryview using the original buffer and updated start/end

(You can't actually do that in MicroPython because we don't support using the slice() constructor, only the subscript syntax, but it works in CPython)

[jimmo]

It's funny I'd never thought about this, but I agree, the terminology is super confusing. 🤣

You slice a cake and now you have a slice of cake. But you wouldn't refer to the desired amount of cake as "the slice".

[ma261065]

Thanks again for the super-clear information. I've been trying to figure this out from various forum posts and the doco for ages. I hope that this level of detail makes it into the doco at some point, but understand that the maintainers have a herculean task updating everything.

[ma261065]

Also referencing #7718 as I think this thread adds extra useful information.

[ondrej1024]

To be able to use the memoryview object, I tried to replace the socket recv() function in my code with the readinto() function.
I am using a blocking socket with timeout and while the recv() function receives the data correctly, the readinto() function times out, even though there is data to read.

This works:
pkt = c.recv(MAX_PACKET_LEN)

This times out:
pkt_sz = c.readinto(buffer[size:],MAX_PACKET_LEN)

What am I missing? Any idea?

[ondrej1024]

To be able to use the memoryview object, I tried to replace the socket recv() function in my code with the readinto() function. I am using a blocking socket with timeout and while the recv() function receives the data correctly, the readinto() function times out, even though there is data to read.

The documentation of read() and readinto() is a bit confusing to me:

Read up to size bytes from the socket. Return a bytes object. If size is not given, it reads all data available from the socket until EOF; as such the method will not return until the socket is closed.

So if the size argument is specified it should read up to this number of bytes, which implies it should stop earlier if there are less bytes available. But it seems to always try to read size bytes and it will block and time out if there are less bytes and don't read anything.