network stuff, memory efficient: sockets, recvfrom into a bytearray

[massimosala]

Hi

I need to read data from some peers, UDP datagrams, using recvfrom.

To avoid to pollute the memory, I wish to use a bytearray

Example:

        client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
	resp = bytearray(_PKT_SIZE)
        while True :
              ... other stuff ...
              client.sendto( ... data ... , host)
              resp[0 : ], addr = client.recvfrom(_PKT_SIZE)
              ... other stuff ...

Using
len(resp)
to get the length of the payload (data received), it always returns _PKT_SIZE !

I cannot use readinto because the peers, during the communication, change their local ports.
I have to use recvfrom to get their new address.
Real example: TFTP

I am not sure about this line:
resp[0 : ], addr = = client.recvfrom(_PKT_SIZE)

Obviously I cannot use
resp, addr = = client.recvfrom(_PKT_SIZE)
because every time it allocates a new instance of resp

Goals:

• keep updated the address of the peers (same ip, their local port keep changing)
• be memory efficient, using a fixed bytearray

Do you have any suggestion?

[robert-hh]

You can use a pre-allocated buffer and memoryview to access it.

[massimosala]

Hi Robert

Sorry, I have to correct myself.
My code using memoryview doesn't work.

mv_resp = memoryview(resp)
while True :
      mv_resp[0 : ], addr = = client.recvfrom(_PKT_SIZE)

Cpython 3.8 error:

memoryview assignment: lvalue and rvalue have different structures

micropython 1.20 error:

lhs and rhs should be compatible

Please show me the correct way to assign the result of recvfrom() to the memoryview and how to get how many bytes received.

[massimosala]

BTW, I hit this problem on the esp8266.
If anyone succeeds, what board are you using?

[bixb922]

This is a way to reuse a bytearray:

# Allocate once when starting
x=bytearray(MAX_SIZE)
...
n=sock.readinto(x)
mydata=memoryview(x)[0:n]

The mydata object will be just a lightweight pointer to the first n bytes of x.
I don't know if this helps, I haven't used readinto for UDP, and I don't know how to retrieve the address in this case.

[massimosala]

Problem solved.

I get the first packet with recvfrom(), and the next packets with readinto(... fixed buffer ...).
It is not elegant and I consume memory with recvfrom(), but it works.

[sosi-deadeye]

We need socket.recvfrom_into().

The reference to CPythons Docs: https://docs.python.org/3/library/socket.html#socket.socket.recvfrom_into

If the received data does not fit in the memoryview, then an CPython exception occurs:

ValueError: memoryview assignment: lvalue and rvalue have different structures

Using the buffer directly, throws a different Exception.

BufferError: Existing exports of data: object cannot be re-sized

socket.recvfrom_into() returns nbytes and the address. It's handy to reuse the same buffer for different tasks, and you know also, how much you have received.

Adding this feature could be used later to optimize asyncio.

[massimosala]

I agree, it will be useful to have a socket.recvfrom_into()

Especially on environments with few memory.

At the moment to implement a TFTP client I have to resort to this method:

• send the request to the server, UDP port 69
• the server will reply switching its local port (per RFC), so I need to get its new address...

So on the client side, receiving
the 1st packet:
data, _host = client.recvfrom(512)
the next packets:
read = client.readinto(mv_buffer[0 :], 512)

It is working, but

• it is sub-optimal, there are two code paths
• 512 bytes (the first packet) are lost until the garbage collector kicks in (and if it will succeed).