can't seem to complete ssl.wrap_socket read method

[RetiredWizard]

I'm running on an Unexpected Maker Feather S2 and MicroPython version 1.19.1
MicroPython v1.19.1 on 2022-06-18; FeatherS2 with ESP32-S2

from os import uname
import sys
import ssl
import usocket as _socket
import network
PORT = 443
HOST = "api.weather.gov"
QUERY = "/gridpoints/BOX/70,76/forecast"
wlan = network.WLAN(network.STA_IF)
if not wlan.active():
    wlan.active(True)
if not wlan.isconnected():
    wlan.connect('myssid','password')
wlan.isconnected()
wlan.ifconfig()[0]
addr = _socket.getaddrinfo(HOST, PORT)[0][4]
client = _socket.socket()
client.connect(addr)
response = ssl.wrap_socket(client)
headers = {"user-agent": "RetiredWizard@"+sys.implementation.name.lower()+uname()[2]}
response.write('GET %s HTTP/1.1\r\nHost: %s\r\nuser-agent: %s\r\n\r\n'%(QUERY,HOST,headers['user-agent']))
print(response.read())

When running the above code the response.read() never returns (at least for several minutes until I ctrl-c). If I give it a read length below 12000 the script completes in a second or two. A length over 13000 hangs and needs to be terminated with a ctrl-c. The actual break point seems to vary somewhere between 12000 and 13000 but I think that has to do with the web site being updated periodically.

The documentation seems to indicate that the read will run until the stream is closed on the server but I'm thinking there should be a way to read all the available output from a website stream which I assume isn't going to close unless it eventually times out.

I tried using select.pool in a loop:

poller = select.poll()
poller.register(response, select.POLLIN)
res = poller.poll(10000)
read_response = ""
while res:
    read_response += response.read(256).decode()
    res = poller.poll(1000)

but the poller.poll in the while loop timed out and the loop ended after a single read.

[RetiredWizard]

I tried running this on a Pico W to make sure it wasn't a board issue and the response.read() seems to behave the same way,

However, the select.poll loop did seem to function properly on the Pico W. The loop I posted to the original question will still hang (unless the stream size is an exact multiple of 256), however by changing the read length from 256 to 1, I do end up with a solution that will completely read the stream, it's not a very fast solution as it does take a while to read the stream one byte at a time.

I then noticed that the other two boards I tested didn't have the latest MicroPython build, so I reflashed them and with the latest nightly build they both behaved the same way as the Pico W. So it looks like the select.poll solution is an option.

I'm still really curious as to whether, I'm going about this completely wrong or if there's another way to read the ssl wrapped stream completely.

[peterhinch]

I'm no expert on this, however this is my take on it. The response from the poller merely tells you that data is available. It doesn't tell you how much to expect. The website response is a fixed length data stream of unknown length. Responding to the poller by reading N bytes will work until you get to the end of the stream. Unless the end of the stream contains exactly N bytes, the read will wait forever (as will read() with no arg). These will only terminate if the server closes the socket, which will never happen.

In other words, if the length of the server response stream is L and your socket reads are of N bytes, the loop will only terminate if L % N is zero. Which is why it works with N == 1.

I'm sure there is a standard solution to this. Off the top of my head one option would be to set a socket timeout if the platform supports it. Another approach is to use nonblocking sockets and uasyncio. You can then apply timeouts to the reading task. An example of this approach is asynchronous MQTT.