another ENOMEM error with PicoW and urequests.post()

[aleppax]

Hi, I'm working with a Raspberry Pico W and 4 sensors connected via i2c.
Before anything else, yes, for more than a month I've tried to dive deep in all the discussions here and in the old forum, and found what seems to me the first and proper solution to ENOMEM errors after requests, and of course I have double checked that every open request is properly closed. But despite this I keep getting this error.
I'm using a fork of micropython with a small change by Felixdoerre #10803 related to a fix for lightsleep.
MicroPython v1.20.0-18-g07a719a31-dirty on 2023-04-27; Raspberry Pi Pico W with RP2040
post.txt
ussl.wrap_socket inside urequests seems the only function, between print markers, that could deal with more memory usage.
attached a log from my pico, the lines without timestamp are print statements added inside the urequests module, to track where the ENOMEM occurs.
I have no clue about how to avoid memory errors, other than deleting all sensors modules after using them and reimporting at the next measuring round, even if it seems not always effective: the log I attached has an error even if during that session no sensor driver has been loaded.
datalogger.py.txt
The data that is going to be sent is 8,5 KB, the memory usage is around 60% and the ENOMEM error is apparently weakly related to the percentage of free memory. Maybe could be related to memory fragmentation? Is it possible to pre allocate enough memory for the request-response objects?
The question is, are there other reasons for this kind of error, where should I look for?
I am strongly motivated to find a solution, should I have to do days of testing.

[scruss]

I know you'll probably need lightsleep for your final version, but does your code (less any lightsleep-reliant bits) work with the stock 1.20 Pico W firmware?

A couple of comments on the code:

• The numbers returned by the gc for heap used/free aren't particularly helpful when you're dealing with fragmentation. You're trying to move a large data structure in a very small system
• In send_data_list, are you sure you're not including multiple copies of the same data, making the data packet larger than it needs to be? I prefer to avoid manipulating json.dumps output, as it sometimes doesn't contain what I've assumed it contains
• My experience of using the WDT on the RP2040 has been more surprising resets/lockups than I'd want

[aleppax]

I've tried, as you suggested, the released firmware, here is the log: apart from the lightsleep which immediately exits, the ENOMEM occurs sometimes but the system recovers afterwards and can post other data (most of the time it succeeds, but this is not a valid metric, sometimes it fails even trying to send only one line instead of 10).
micropython-vanilla.log

• gc: I had doubts about this, and indeed you confirm them. Anyway I've found that the micropython.mem_info(1) prints a visual representation of the allocated memory, so far it seems to be one of the most useful tools to better understand the situation.
• I don't know, I trust the integrity of the data manipulated by micropython and since among the data collected there is also a timestamp I would have noticed redundant data
• It was a bit of a bloodbath for me too, but then with Felixdoerre's fix I had no more problems. I don't know whether it's related, but usually after a while the communication with Thonny ends, but this doesn't affect the normal behavior of the system, it continues to work and to log.

In the meanwhile I would have tried the PR #8968 but it seems that is missing the part that in Cpython preallocates memory for the SSLcontext, moreover I've still not understood the usage for a client application.
If confirmed (no memory preallocation), as a temporary solution I will fall back on the insecure http, (after all, the risk of compromising these measures is ridiculous) waiting for developments in the code.
thank you Stewart for your suggestions.

I am sending data with insecure HTTP, transmission speed is at least 20 times higher! a nice battery saver considering that the WiFi consumes a lot. all in all the system gained something.

[Carglglz]

@aleppax I would say those ENOMEM errors are definitely SSL related, creating a ssl socket with every https request doesn't play out very well as you already tested. As an alternative you should consider using mqtt, which works with SSL too and avoids these memory allocations errors since it only needs to be connected once.

[aleppax]

This is good to know since we already use mqtt as an optional channel, but honestly I haven't tested it extensively. Thank you!

[smudge1977]

try calling this free(True, True) on your main loop.
I found using the urequests lib I had all my memory disapear if I do not call the gc.collect() every loop

import gc

def free(full=False, collect=False):
    if collect:
        gc.collect()
    F = gc.mem_free()
    A = gc.mem_alloc()
    T = F+A
    P = '{0:.2f}%'.format(F/T*100)
    if not full:
        return P
    else:
        return ('Total:{0} Free:{1} ({2})'.format(T, F, P))

[smudge1977]

Note even with an empty program you seem to "loose" memory:

import gc
import os
import time


def df():
    s = os.statvfs('//')
    return ('{0} MB'.format((s[0]*s[3])/1048576))


def free(full=False, collect=False):
    if collect:
        gc.collect()
    F = gc.mem_free()
    A = gc.mem_alloc()
    T = F+A
    P = '{0:.2f}%'.format(F/T*100)
    if not full:
        return P
    else:
        return ('Total:{0} Free:{1} ({2})'.format(T, F, P))

while True:
    print(free(True, False))
    time.sleep(1)

gives this:

Total:177280 Free:170000 (95.89%)
Total:177280 Free:169776 (95.77%)
Total:177280 Free:169552 (95.64%)
Total:177280 Free:169328 (95.51%)
Total:177280 Free:169104 (95.39%)
Total:177280 Free:168880 (95.26%)
Total:177280 Free:168656 (95.14%)
Total:177280 Free:168432 (95.01%)
Total:177280 Free:168208 (94.88%)
Total:177280 Free:167984 (94.76%)
Total:177280 Free:167760 (94.63%)
Total:177280 Free:167536 (94.50%)