Cannot use urequests library

[FabriceAuz]

Hi
I'm testing the urequests library, to later launch requests to a weather API.
I installed it with Thonny, and Thonny says (sorry it's in French):

Version installée : 0.1.2
Installée dans : c:\users\fa125436\appdata\roaming\thonny\plugins\python310\site-packages

Dernière version stable: 0.1.2
Résumé: HTTP Library with a similar interface to python-requests. Particularly well suited for usage with MicroPython.org
Auteur: Alex Cowan
Page d'accueil: https://github.com/lucien2k/wipy-urllib
Page PyPI: https://pypi.org/project/urequests/

However, when I try this simple skecth:

import urequests  #                       <-- this is line 142
url = "http://example.com"
# Perform HTTP GET request on a non-SSL web
response = urequests.get(url)  #         <-- this is line 145
# Display the contents of the page
print(response.text)

I immediately get the error message:

Connection successful
('192.168.1.10', '255.255.255.0', '192.168.1.1', '8.8.8.8')
Traceback (most recent call last):
File "", line 142, in
ImportError: no module named 'urequests'

I had to copy the urequests.py file from here on the ESP32 flash to bypass this error. But, then I get another error:

Already connected
Traceback (most recent call last):
File "", line 145, in
File "urequests.py", line 104, in get
File "urequests.py", line 124, in urlopen
File "urequests.py", line 25, in init
OSError: -202

Line 25 is :
addr = socket.getaddrinfo(host, int(port))[0][-1]
and the parameters seem correct: host is 'example.com', port is 80.

So, what is going wrong?
Thanks for your help

[jimmo]

Confusingly.. this is not the official urequests library.

Simple answer though, your ESP32 firmware should already have the urequests library installed, there's no need to install it manually.

Longer answer -- the PyPI package namespace is completely disorganised and difficult to navigate, as well as difficult to publish micropython-specific packages to. We no longer support it as a source for installing packages. Thonny is a third-party tool.

The official version of urequests is in micropython-lib. You can install packages from micropython-lib using mip (either on-device on using mpremote). (Note that on-device is only supported right now on the nightly builds and will be in the upcoming v1.20 release). https://docs.micropython.org/en/latest/reference/packages.html

[FabriceAuz]

Thanks @jimmo
The ESP32 is an ESP32-C3, part of a T-Pico C3 board. I used the official firmware from the Micropython website, here.
However, as I said above, urequests does not seem to be included:

import urequests
Traceback (most recent call last):
File "", line 1, in
ImportError: no module named 'urequests'

When I try mip, here is what I get:

import mip
Traceback (most recent call last):
File "", line 1, in
ImportError: no module named 'mip'

There are 2 urequests.py files in the Github:

• This one does not have the get, post, request methods
• The other one has them, but when I copy this file in the ESP32 and run my sketch, I get a new error:

Traceback (most recent call last):
File "", line 140, in
File "GetData.py", line 47, in getWeather
File "urequests.py", line 123, in request
TypeError: object with buffer protocol required

My function does this:

def getWeather():
    # First request : today + tomorrow, all data
    now = utime.localtime()
    today = "%d-%d-%dT%d:%d:%d-01:00"%now[:-2]
    nowP1 = utime.localtime(utime.mktime((now[0], now[1], now[2]+1, now[3], now[4], now[5], now[6], now[7])))
    tomorrow = "%d-%d-%dT%d:%d:%d-01:00"%nowP1[:-2]
    url = "https://api.tomorrow.io/v4/timelines"
    
    fields = [
      "precipitationIntensity",
      "precipitationType",
      "humidity",
      "windSpeed",
      "windDirection",
      "temperature",
      "precipitationProbability",
      "cloudCover",
      "weatherCode",
      "sunriseTime",
      "sunsetTime",
      "moonPhase",
      "uvIndex",
    ]
    # timesteps = ["current", "1h", "1d"]
    
    querystring = {
    "location":"48.730756, 2.27137",
    "fields":fields,
    "units":"metric",
    "timesteps":"1d",
    "startTime":today,
    "endTime":tomorrow,
    "apikey":myKEY}
    
    headers = {"accept": "application/json"}
    response = urequests.request("GET", url, data=querystring)  # <-- line 47

What is wrong with my call?
It seems the problem is in the data , maybe wrong type or wrong form?

[jimmo]

However, as I said above, urequests does not seem to be included:

I don't completely understand why this is wasn't included in v1.19.1 release for the C3 -- I think there's something about how the release builds used to work and I know this was fixed after v1.19.1. Anyway, I recommend using the nightly build instead then, it definitely has urequests in it (I checked).

• This one does not have the get, post, request methods

Confusingly, that's "urllib.urequest" (no "s" on the end).

• This one does not have the get, post, request methods

This is the right one.

It seems the problem is in the data , maybe wrong type or wrong form?

That's right. Unfortunately you've run into micropython/micropython-lib#534 (comment) -- urequests (because it's "micro") doesn't implement every feature of "requests", and one of those features is automatically converting data as a dictionary of key/value pairs into a URL-encoded payload. You have to do this yourself.

The error you're getting is because it's trying to use data directly as a buffer (i.e. a string or bytes to send directly to the socket).

Like I said in the comment, I think this is probably something we should consider adding to urequests.

When I try mip, here is what I get:

This is what I was saying above "Note that on-device is only supported right now on the nightly builds and will be in the upcoming v1.20 release"

[FabriceAuz]

Thanks a lot for your help, I'll have a look at that very soon

[FabriceAuz]

So I used the urllib.urequests - it seems you're the author?

response = urequests.urlopen(url, url_escape(querystring), "GET")

But I don't know how to extract the useful values from the payload.
When I print response, I get:
<_SSLSocket 3fca72c0>
and it has no 'json' attibute.

How can I extract the values from the response?

Also, when I uncomment the print(l) lines in your code, I get the following:

[b'HTTP/1.1', b'401', b'Unauthorized\r\n']
b'Date: Sat, 29 Oct 2022 13:40:57 GMT\r\n'
b'Content-Type: application/json; charset=utf-8\r\n'
b'Content-Length: 124\r\n'
b'Connection: close\r\n'
b'WWW-Authenticate: Key realm="kong"\r\n'
b'vary: Origin\r\n'
b'Access-Control-Allow-Origin: *\r\n'
b'X-Moesif-Transaction-Id: a577cd35-b47a-4393-accb-3bb8ce2d75aa\r\n'
b'X-Kong-Response-Latency: 8\r\n'
b'Via: 1.1 google\r\n'
b'CF-Cache-Status: DYNAMIC\r\n'
b'Server-Timing: cf-q-config;dur=6.0000002122251e-06\r\n'
b'Server: cloudflare\r\n'
b'CF-RAY: 761c53748c41d712-CDG\r\n'

but I don't know what it means...

b'Unauthorized

Seems bad to me. But I have successfully done the same request using python on my laptop.

[FabriceAuz]

It seems the url_escape function I copied from the abovementionned issue does not act as expected. I'll investigate

def url_escape(s): # Translate payload to URL-encoded
    return ''.join(c if c.isalpha() or c.isdigit() else '%%%02x' % ord(c) for c in s)

querystring = {"location":"48.730756, 2.27137","fields":fields}
url_escape(querystring)
Out[10]: 'locationfields'

[jimmo]

Ah I see -- yeah this is URL escaping, not querystring encoding.

You want something like

def url_escape(s): # Translate payload to URL-encoded
    return ''.join(c if c.isalpha() or c.isdigit() else '%%%02x' % ord(c) for c in s)

def url_querystring_encode(params):
  return "&".join("{}={}".format(url_escape(n), url_escape(v)) for n, v in params.items())

Then your url to pass to urequests.get(url) should be

querystring_params = {
"location":"48.730756, 2.27137",
    "fields":fields,
    "units":"metric",
    "timesteps":"1d",
    "startTime":today,
    "endTime":tomorrow,
    "apikey":myKEY
}
url = "https://api.tomorrow.io/v4/timelines?" + url_querystring_encode(querystring_params)
response = urequests.get(url)

# now you can use response.json

[jimmo]

it seems you're the author?

No I'm just a maintainer.

But I don't know how to extract the useful values from the payload.
When I print response, I get:
<_SSLSocket 3fca72c0>
and it has no 'json' attibute.

urllib.urequest is very simple -- it just gives you back the socket containing the HTTP response payload. See https://docs.python.org/3/library/urllib.request.html#urllib.request.urlopen

I think you can probably do

import json
response_json = json.load(response)

But just use urequests, it supports getting the response as json.

but I don't know what it means...

This is the HTTP response headers. It's telling you the request was unauthorised (I suspect because the API key was not correctly specified).

I'm surprised to see a data payload being specified for a GET request (this would normally only be used for a POST). You should instead be specifiying the query string on the actual URL.

The documentation for the API you're using (https://docs.tomorrow.io/reference/get-timelines) provides a Python example on the right hand side that should work:

# Copied from https://docs.tomorrow.io/reference/get-timelines
import requests

url = "https://api.tomorrow.io/v4/timelines?location=42.3478%2C%20-71.0466&fields=temperature&units=metric&timesteps=1h&startTime=now&endTime=nowPlus6h"

headers = {
    "accept": "application/json",
    "Accept-Encoding": "gzip"
}

response = requests.get(url, headers=headers)

print(response.text)

So to adapt that, change the first line to import urequests as requests and make the url be calculated using your url_escape(querystring) function, i.e. "https://api.tomorrow.io/v4/timelines?" + url_escape(querystring)

(Where did you get url_escape from, as long as it's generating the escaped name=value&name=value&... format then it should be fine.

[FabriceAuz]

Thanks, but I don't see where to put the API key and the associated syntax.
Got it! apikey=blahblah&...

Many thanks for your help, I'll try this soon and keep you updated.

[FabriceAuz]

Getting better, I now have the response, but can't process it.
When I print it I get this:
b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\xff\xb4\x91=o\xc20\x10\x86\xff\xcb\xcd!\xf2\x07\xa4\xe6V\xba\xb0!\x95\xa9\x15\x83\xc1\'a)q"\x7f\xa4E\xc8\xff\xbdJ(C\xa3\xb4\xa2CG\xbf\xf7Z~\xee\xf1\x15\x8c\x8e\x1a\xf0\n\xd16T[G\x01\xf0\xedv\n\x91:@\xe0\x06\n g\xf6\xb6!@\x10L\x88\x05g\x0b\xc9\xf6l\x89\x8c!c\xafP@\x88\xda\xc7IE\xac\xbfU\xac\x8b\xe4{]\xdf^x\xe0B\xaf\xeb4\xf0\\\xe1T\xb7\xc9l\xda\x9e< g\xac\x80sj\xac\xb1\xf1\x02\xa8\xaar\xb5*\xa0i[\xb7;\xeb@\x80\xbc\x80\xce\xd3\xc9v6\xeah[\xb7u\x91\\\x18\xcbl2\xda\xf9\xf6\xa8\x8f\xb6\x9e\x1d\xee/\x1d\x8diH\xce\xdb@3\xb0\x15\x8a\xf5]@r\x81f\x16\xe2\x15\xca{\'R\xd3\x91\xd71y\x02\x14\xb2T\xaa\x80\xd4o\x9d\xa1\x8f\x11\xfb\x9dt<\x93\xdf\xb4\x86\xc6=\x87\xc8:\xf3l=\x9d\x06"@\xae\x96\xa5\xfa\x8a_:"\x03(\xcbJ\xe6\\\xcc\x1b\x9d\xfc\xd2cF\xd5\x08\xf6oF\xf9OF\x07\xd8\n%\xfb\xcd\xa8d\xa3Q5g\x94\xafK\xf9W\xa1B>\x95\xd5\xc4\xa7P9\x1f\xf2!\xe7\xcf\x00\x00\x00\xff\xff\x1d4\x03\xd0\x1e\x03\x00\x00'

It's not utf-8, so .decode('utf-8') doesn't work.

print(response.text()) leads to an error:

Traceback (most recent call last):
File "", line 140, in
File "GetData.py", line 66, in getWeather
File "urequests.py", line 28, in text
UnicodeError:

and print(response.json()) as well:

Traceback (most recent call last):
File "", line 140, in
File "GetData.py", line 67, in getWeather
File "urequests.py", line 33, in json
ValueError: syntax error in JSON

How can I decode this?

Here is the answer from the website (when I paste the url from my micropython code):

{"data":{"timelines":[{"timestep":"1d","endTime":"2022-10-30T04:00:00Z","startTime":"2022-10-29T04:00:00Z","intervals":[{"startTime":"2022-10-29T04:00:00Z","values":{"cloudCover":100,"humidity":86.55,"moonPhase":1,"precipitationIntensity":0,"precipitationProbability":0,"precipitationType":0,"sunriseTime":"2022-10-29T06:29:00Z","sunsetTime":"2022-10-29T16:39:00Z","temperature":23.88,"uvIndex":1,"weatherCode":1001,"windDirection":184.81,"windSpeed":3.63}},{"startTime":"2022-10-30T04:00:00Z","values":{"cloudCover":100,"humidity":88.88,"moonPhase":1,"precipitationIntensity":0,"precipitationProbability":0,"precipitationType":1,"sunriseTime":"2022-10-30T06:30:00Z","sunsetTime":"2022-10-30T16:38:00Z","temperature":19.3,"uvIndex":1,"weatherCode":1001,"windDirection":238.55,"windSpeed":3.28}}]}]}}

[FabriceAuz]

I found the problem :
headers = {"accept": "application/json", "Accept-Encoding": "gzip"}
instead of
headers = {"accept": "application/json", "Accept-Encoding": "utf-8"}
Now it works and I get a json response.

[GM-Script-Writer-62850]

you can use gzip, but you would need to decompress it (not sure how to do that on a pico), doubt that is worth the overhead unless you are using dial up

[sameer-98]

Did you figure out how to solve the OS 202 error