Strange behaviour with requests / Memory Issue?

[klauweg]

After a few test requests from REPL i get this error:

>>> a=requests.get('https://blog.codelix.de/rss.xml')`
>>> a=requests.get('https://blog.codelix.de/rss.xml')
>>> a=requests.get('https://blog.codelix.de/rss.xml')
>>> a=requests.get('https://blog.codelix.de/rss.xml')
>>> a=requests.get('https://blog.codelix.de/rss.xml')
>>> a=requests.get('https://blog.codelix.de/rss.xml')
>>> a=requests.get('https://blog.codelix.de/rss.xml')
>>> a=requests.get('https://blog.codelix.de/rss.xml')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/__init__.py", line 180, in get
  File "requests/__init__.py", line 93, in request
OSError: [Errno 1] EPERM

This error is reproducible and happens after about 7-9 requests.
Sometimes the following memory allocation error is printed:

E (30906) esp-sha: Failed to allocate buf memory
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/__init__.py", line 93, in request
OSError: [Errno 1] EPERM

After doing gc.collect() the request works again. But still only about 7 times.

This is strange because micropython is running on ESP32 with 8MB PSRAM:

(sysname='esp32', nodename='esp32', release='1.22.2', version='v1.22.2 on 2024-02-22', machine='Generic ESP32S3 module with Octal-SPIRAM with ESP32S3')

The reported free mem does not vary much and the memory is far away from being exhausted.
This memory allocation error is reported anyway and it works again after gc.collect().
Here you can see with mem_free() values:

>>>
>>> gc.collect()
>>> gc.mem_free()
8184976
>>> a=requests.get('https://blog.codelix.de/rss.xml')
>>> a=requests.get('https://blog.codelix.de/rss.xml')
>>> a=requests.get('https://blog.codelix.de/rss.xml')
>>> a=requests.get('https://blog.codelix.de/rss.xml')
>>> a=requests.get('https://blog.codelix.de/rss.xml')
>>> a=requests.get('https://blog.codelix.de/rss.xml')
>>> a=requests.get('https://blog.codelix.de/rss.xml')
E (426666) esp-sha: Failed to allocate buf memory
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/__init__.py", line 180, in get
  File "requests/__init__.py", line 93, in request
OSError: [Errno 1] EPERM
>>> a=requests.get('https://blog.codelix.de/rss.xml')
E (444686) esp-sha: Failed to allocate buf memory
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/__init__.py", line 180, in get
  File "requests/__init__.py", line 93, in request
OSError: [Errno 1] EPERM
>>> gc.mem_free()
8154224
>>> gc.collect()
>>> a=requests.get('https://blog.codelix.de/rss.xml')
>>> a=requests.get('https://blog.codelix.de/rss.xml')
>>> a=requests.get('https://blog.codelix.de/rss.xml')
>>> a=requests.get('https://blog.codelix.de/rss.xml')
>>> a=requests.get('https://blog.codelix.de/rss.xml')
>>> a=requests.get('https://blog.codelix.de/rss.xml')
>>> a=requests.get('https://blog.codelix.de/rss.xml')
E (476876) esp-sha: Failed to allocate buf memory
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/__init__.py", line 180, in get
  File "requests/__init__.py", line 93, in request
OSError: [Errno 1] EPERM
>>> gc.mem_free()
8156912
>>>

Did i do something wrong or is this a bug?

[klauweg]

I have discovered micropython.mem_info() which is more detailed than gc.mem_free():

>>> gc.collect()
>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 64000, used: 5456, free: 58544, max new split: 8126464
 No. of 1-blocks: 78, 2-blocks: 13, max blk sz: 98, max free sz: 1392

After the usual 7 request() calls:

>>> a=requests.request("get",'https://www.tagesschau.de/infoservices/alle-meldungen-100~rss2.xml')
>>> a=requests.request("get",'https://www.tagesschau.de/infoservices/alle-meldungen-100~rss2.xml')
E (2197926) esp-aes: Failed to allocate memory
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/__init__.py", line 125, in request
OSError: -1
>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 64000, used: 44816, free: 19184, max new split: 8126464
 No. of 1-blocks: 919, 2-blocks: 133, max blk sz: 98, max free sz: 987

I'm not familiar with the micropython memory management, but as far as i understand the "total" value should automatically expand with new splits from real total memory (8MB). This works with something like a=5000000*"a" but apparently not with https requests. Reading a little bit code it may be related to ssl. But i really don't understad all this stuff :-).

[bixb922]

You'll need to .close the response object returned to free the associated socket and it's memory

[klauweg]

That makes sense.
So the limit is not the memory but the associated sockets?
And if i call gc.collect() manually the sockets are freed also?

[scruss]

only if you close the response object

[klauweg]

In my experience gc.collect() releases the associated socket also. Even without .close().
But of course this is not reasonable.

[glenn20]

Socket objects have a __del__() dunder method which will automatically close the socket before freeing the socket object.

However, you are then relying on gc to control your socket resources. It is better to be explicit and close the socket when you are finished with it.

[klauweg]

Yes, of course.
I was just trying to understand what was happening.
Thank you!