Memory allocation issues with urequests.get()

[jsa91]

Hi,

I'm using a WROOM32 to fetch and display electricity prices on a ILI9341 display. I'm trying to fetch the electricity prices from this URL in .json format:
https://www.elprisetjustnu.se/api/v1/prices/2023/12-28_SE3.json

Since I'm running both wifi and display driver on the esp32 I seem to be running out of memory with urequests.get()

Firmware: ESP32_GENERIC-20231227-v1.22.0.bin

import urequests
import micropython

micropython.mem_info()
response = urequests.get("http://www.elprisetjustnu.se/api/v1/prices/2023/12-28_SE3.json")

stack: 976 out of 15360
GC: total: 111936, used: 106400, free: 5536, max new split: 31744
 No. of 1-blocks: 761, 2-blocks: 132, max blk sz: 2400, max free sz: 71

Traceback (most recent call last):
  File "main.py", line 30, in <module>
  File "price.py", line 24, in get_prices
  File "requests/__init__.py", line 180, in get
  File "requests/__init__.py", line 163, in request
  File "requests/__init__.py", line 93, in request
OSError: [Errno 12] ENOMEM

How should I approach this problem? Since I'm trying to fetch .json formated text, is it possible to fetch only part of the .json and not everything at once? Should i try to implement gc.collect() somewhere in the code?

[shariltumin]

You do not have to use urequests.get(). And you must use https to connect to the server, otherwise you will get "301 Moved Permanently".

You may want to try the following script:

import time, network, socket, ssl

# Connect to WiFI
network.WLAN(network.AP_IF).active(False)
wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.config(pm=0) # disable power management
wlan.config(txpower=20)
wlan.connect("YOUR_SSID", "YOUR_PWD")
time.sleep(5)
print(wlan.ifconfig())

srv = 'www.elprisetjustnu.se' 
port = 443 # https port number
addr = socket.getaddrinfo(srv, port)[0][-1]

# Connect to htps
ss = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ss.connect(addr)
time.sleep(1)
s = ssl.wrap_socket(ss)

# Send request and get reply
s.write(b'GET /api/v1/prices/2023/12-28_SE3.json HTTP/1.1\r\nHost: www.elprisetjustnu.se\r\n\r\n')
data = s.read()
for d in data.decode().split('\r\n'):
    print(d)

# Close sockets
s.close()
ss.close()

# Done!

Hope this helps.

[ctimmer]

shariltumin - Thanks for the code, I will save this for future reference.

I ran the request code on my picow with this result (same for http and https):

MicroPython v1.22.0 on 2023-12-27; Raspberry Pi Pico W with RP2040
Type "help()" for more information.
stack: 572 out of 7936
GC: total: 192000, used: 6432, free: 185568
 No. of 1-blocks: 73, 2-blocks: 12, max blk sz: 64, max free sz: 11457
[{"SEK_per_kWh":0.33258,"EUR_per_kWh":0.0301,"EXR":11.04907,"time_start":"2023-12-28T00:00:00+01:00","time_end":"2023-12-28T01:00:00+01:00"},...

Using the 'read' function is there a possibility part of the message could be missed if the response is broken up in to 2 or more messages? I may be showing my ignorance here.

The memory usage in the original post seemed really high, Using 'gc' after all of the initialization might help to fix the problem.

Curt

[shariltumin]

You are right, both http and https gave the same results. There's some magic in "requests" that I'm not aware of. I have learned something new. I need to study the code of the "requests" module. The "301 Moved Permanently" statement was from unix curl.

I do not know what will happen if the server sends the response in chunks. socket.read() will read without error as long as the server keeps its socket end open.

[jsa91]

Thank you for you reply @shariltumin , your script worked great. Unfortunately I still have memory allocation issues if I run your script after setting up the graphical interface for the ILI9341 display.

Fetching from URL: www.elprisetjustnu.se
Traceback (most recent call last):
  File "main.py", line 49, in <module>
  File "price.py", line 35, in get_prices
MemoryError: memory allocation failed, allocating 3407 bytes

I have also tried to invoke gc.collect() after setting up the display, right before calling my class for fetching the prices:

micropython.mem_info()
gc.collect()
micropython.mem_info()
api = ElectricityPriceAPI()

# Output:
stack: 656 out of 15360
GC: total: 111936, used: 108176, free: 3760, max new split: 22528
 No. of 1-blocks: 1443, 2-blocks: 113, max blk sz: 2400, max free sz: 189

stack: 656 out of 15360
GC: total: 111936, used: 92960, free: 18976, max new split: 22528
 No. of 1-blocks: 542, 2-blocks: 88, max blk sz: 2400, max free sz: 388

Here is my class for fetching the prices:

import config
import utime
import time 
import socket
import ssl

class ElectricityPriceAPI:
    def __init__(self):
        self.ntp_time = utime.localtime()
        
    def get_prices(self, day):
        
        if day == "today":
            day_target = self.ntp_time
        elif day == "tomorrow":
            day_target = self.ntp_time(utime.time() + 86400)

        year, month, date = "{:04}".format(day_target[0]), "{:02}".format(day_target[1]), "{:02}".format(day_target[2])

        addr = socket.getaddrinfo(config.url, config.port)[0][-1]
        print(f"Fetching from URL: {config.url}")

        # Connect to https
        ss = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        ss.connect(addr)
        time.sleep(1)
        s = ssl.wrap_socket(ss)

        request = f"GET {config.api}{year}/{month}-{date}_{config.zone}.json HTTP/1.1\r\nHost: {config.url}\r\n\r\n"
        s.write(request.encode())
        data = s.read()
        price = data.decode().split('\r\n')
        
        s.close()
        ss.close()
        return price

Is it possible that the display driver is allocating to much memory for my project?
I am using bits and pieces from:
https://github.com/peterhinch/micropython-nano-gui

[shariltumin]

Note that ssl uses memory when in use. So that could be the problem here.
I assume you only fetch the spot prices once a day? I will try this way:

1. get the price data from the URL (using ssl!)
2. store the price data to a file
3. delete all ssl, socket objects
4. gc.collect
5. import display driver
6. read price data from the file

This way you will not be using ssl when memory is tight. It is worth a try if your project allows you to do things in parts like the above.

[jsa91]

Yep, that solved it. Thank you!