UDP packet loss when memory is fragmented

[jfcliche]

I have implemented on my ESP32C3 a simple SSDP discovery function that multicasts a UDP requests and receives unicast responses from all servers on the subnet. UDP responses are 400-ish bytes long and are coming in a very rapid burst of 3 packets, sometimes separated by a few microseconds.

I observe that if my memory is heavily used and is fragmented, I almost systematically lose the 3rd UDP packet.
I have tried reading my packets in a tight loop with socket.recv(), socket.read(), socket.readinto(), blocking socket or not, with or without select.poll(), but I always have the same packet loss. This does not happen if I load only the discovery module and not the rest of the application.

To me it looks like the UDP packets are arriving faster than I can process them and the underlying LWIP code cannot allocate PBUF space for it and just silently drops the packet. Is that expected? I would not have thought that LWIP was sharing heap memory with Python, but there seems somehow to be a link with python memory utilization. I thought that maybe the garbage collector kicks in more often and slows down things enough to lose packets, but calling gc.collect() before the packet receiving loop does not help.

Any thoughts?

[sophiedegran]

Hi
I would suggest you to look at the asyncio methods for that with StreamReader and StreamWriter, also as you have many exchange it seemed you have to Queue.
Look at Peter example

[jfcliche]

@sophiedegran thanks for the pointer . I found the examples and tried to use StreamReader. I have the following issues

1. I use UDP , not TCP . I might be mistaken, but StreamReader does not seem to work with UDP sockets. I don't get any data.
2. As per the SSDP protocol, I send packets to a multicast address but receive them from a unicast address, so I can't just use socket.connect() to establish a bidirectional connection.

The normal socket.read() or socket.recv() would work just fine if the UDP stack did not drop packets. I have tried socket.read() with a buffer size large enough to get all 3 packets, but I only get 2 once I imported a few more modules in python. I have the feeling that the problem is at a lower level than micropython top level socket commands.

[peterhinch]

You are right. StreamReader is not suitable for UDP.

[bidrohini]

Use esp_get_free_heap_size() and esp_get_minimum_free_heap_size() to monitor heap availability during the burst. Check if the PBUF pool size or allocation failures correlate with packet loss.

[jfcliche]

I have not found how I can access these exact IDF functions form micropython but that put me in the right path. I do have access to esp32.idf_heap_info, which seems to give the best clue on what is going on.

This is the memory situation when I receive all 3 UDP packets correctly:

>>> esp32.idf_heap_info(0)
[(128912, 4, 0, 4), (116496, 108296, 106496, 4), (10576, 312, 288, 24), (6072, 4188, 4096, 780)]
>>> micropython.mem_info()
stack: 484 out of 15360
GC: total: 64000, used: 25840, free: 38160, max new split: 106496
 No. of 1-blocks: 666, 2-blocks: 57, max blk sz: 64, max free sz: 2122
 >>> gc.mem_free()
141920

This is what I have after I imported a bunch of additional modules, which is also when I systematically lose the 3rd UDP packet of each burst:

>>> esp32.idf_heap_info(esp32.HEAP_DATA)
[(128912, 4, 0, 4), (116496, 1540, 1344, 4), (10576, 312, 288, 24), (6072, 780, 768, 780)]
>>> micropython.mem_info()
stack: 484 out of 15360
GC: total: 167872, used: 134240, free: 33632, max new split: 1344
 No. of 1-blocks: 1411, 2-blocks: 208, max blk sz: 2560, max free sz: 1281
>>> gc.mem_free()
34816

At that point, Micropython seemed to have allocated memory from the 2nd and 4th pools, and the maximum available allocation (3rd number of each tuple) is down to 1344 bytes. Not enough for a MTU=1500 bytes packet buffer. Are any of these pools also used for PBUFs?

I played a bit with this and allocated bytearrays 10K increments until the 2nd pool max allocatable size was 1600 bytes. Pool 4 was unaffected. Then I ran my packet capture function, which silently failed receiving the 3rd UDP packet. After that even the 4th pool had been allocated from and was also low on resources. So there seems to be a correlation between free memory in heap pool 2 and/or 4 and UDP packet loss.

I admire Micropython for its tenacity in trying to scrape memory resources from wherever it can to keep the program running. Is there a way to prevent it from drawing too much from a particular heap (without recompiling it)? It would be nice to be able to ensure a deterministic UDP packet processing performance independent from the dynamics of Micropython memory allocation.