Micropython's urequests behaves differently than python's requests #10946

0slb · 2023-03-06T17:49:58Z

0slb
Mar 6, 2023

Hi,

I am using Micropython's urequests module to access a website but it returns 403 Access Denied error. However, when I use Python's requests module to access the same website, it returns the actual content of the website.

When using urequests:

import urequests

headers={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'}

url="https://www.zara.com"

response = urequests.get(url, headers=headers)

print(response.content)

Returns:
b'<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don\'t have permission to access "http://www.zara.com/es/es/blazer-traje-estructura-p00706334.html?" on this server.<P>\nReference #18.c4d31102.1677369242.52175e1f\n</BODY>\n</HTML>\n'

When using Python's requests:

import requests

headers={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'}

url="https://www.zara.com"

response = requests.get(url, headers=headers)

print(response.content)

It returns the actual content of the website.

I've tried looking what could cause this problem but haven't been able to make urequests behave like requests.

davefes · 2023-03-06T18:48:31Z

davefes
Mar 6, 2023

Just to help narrow the search ... which device? I recall a thread that involved Microdot as a webserver and this issue. Try searching the old forum. Actually, there is a page of hits here.

5 replies

0slb Mar 6, 2023
Author

I'm using micropython on a Raspberry Pi Pico W.

Josverl Mar 6, 2023
Collaborator Sponsor

Could it be that the website you are accessing uses redirects http 30x ?

If I remember correctly these are not supported by urequests

Recycling old bits:
https://forum.micropython.org/viewtopic.php?t=8152#:~:text=Note%20that%20urequests%20doesn%27t%20support%20redirects%20%28although%20see,Request%20-%20Access%20Google%20Apps%20Script%20Web%20API

0slb Mar 8, 2023
Author

Hi,

I've done a little bit of research and from what I've read using curl -i returns the response of the request. If for example when trying to access https://spotify.com/ it does indeed retreive a 30x redirect (301). But when requesting zara.com it returns 403 and when send an user-agent, using: curl -A "Mozilla/5.0 (iPad; CPU OS 13_2 like Mac OS X)" -i https://www.zara.com/ it returns the actual web page and OK response 200.

So, from what I know it doesn't look like it's trying to redirect.

Thanks

Josverl Mar 9, 2023
Collaborator Sponsor

And which URL did you try for the original question?

That is relevant as the server / webapp determines what response to return. And apart from very simple sites, that will vary, per scenario, header, cooky and path

0slb Mar 10, 2023
Author

It's true that .com and .com/ aren't the same, though doing curl -A "Mozilla/5.0 (iPad; CPU OS 13_2 like Mac OS X)" -i https://www.zara.com/ and curl -A "Mozilla/5.0 (iPad; CPU OS 13_2 like Mac OS X)" -i https://www.zara.com they return the same:

HTTP/2 200
content-type: text/html
content-length: 2215
date: Fri, 10 Mar 2023 22:48:44 GMT
set-cookie: ITXSESSIONID=1bec16e6f074a2a7b51b0637b5ec4aa8; expires=Fri, 10-Mar-2023 23:18:44 GMT; path=/; domain=.zara.com; secure
server-timing: cdn-cache; desc=HIT
server-timing: edge; dur=1
server-timing: ACTT;dur=1,ACRTT;dur=93
cache-control: no-store, private
strict-transport-security: max-age=31536000
expires: 0
cache-control: no-cache, no-store, must-revalidate
set-cookie: _abck=EE9E89FBA8A97B54691D06D814EC26E1~-1~YAAQrSURAk6TEb6GAQAANVa2zQmwSP59Aq15B41+79h7XCmO9fMyFgR3mIQ1HA1IQJj7I0IixJhcnPVKgfsPo9xGnMiMCTr/2zZ1m19S2ScE/8/1xS5HNjNSBY5obvuRCXVqTtYBS+LUnAs6DDi40RvHXEgurhMzPg3qpSs9Sx5RjJMOgFqlqz/TOqozyqHlcukB3FU6Vy0FXs/QFEIr7YUCH8Z45ekZ76pOScwjVtn1pDFHb5Cxrqv/Om3UedLENXHEz/xcckuAK0vSfko6gV/GqZJdmRtPEDh8cowM2RB12/FDtz1XiPlMtK453n9rIQyZ0beQxBZBRoZXmfIzbsEdVmxhxjbAxhQQffH4F7P0PDoYgP7A6YFG5Q==~-1~-1~-1; Domain=.zara.com; Path=/; Expires=Sat, 09 Mar 2024 22:48:44 GMT; Max-Age=31536000; Secure
set-cookie: ak_bmsc=BC0E013DFFF22115852DAEF7E3E78A5E~000000000000000000000000000000~YAAQrSURAk+TEb6GAQAANVa2zRMUWb0KNntIfXXEK/1QjeK4+fMeITDu2DTMe8YWYPWjt5QEOG1v5Dv8rgmVStTAEA/pkQW8WfpijAYhBdbOHyQQ4HMReRfzBG58QwgURmW71MnDaa/2Z/GvEBff1c6orS55gHlYK1JdDf6V58soGoLYOXs3exBYUTffNqQPlrS4zCeHraSx0C0y7hMqeu9HbnzDwcbVVJyY+wnxhjnf37vEZBWxO3NiFveeICCOY+vx69zJZImzBODu5LT0bHcYqDabDH1jX3wCViNpLNTVHLYDS653gv0JQYvq8n0QrSUgtFFolqHKWDKWMszgzJmIDZ3XJsb6C4xImItNyu+MeVJzm5wAFJe4xyZwcBAX7ft5OrKFIXFe6Qe8SxP1WX1BPg4WQHLvKxw0sTHq4UCc; Domain=.zara.com; Path=/; Expires=Sat, 11 Mar 2023 00:48:44 GMT; Max-Age=7200; HttpOnly
set-cookie: bm_sz=44A7DF621DE92883A7B19F24EC7268BC~YAAQrSURAlCTEb6GAQAANVa2zRMoWq3nqcY6v1h4XkSrtF57v/34U2sa+c2F4ibxpcBRkOdchVsQUvw77Ceqhz1XRf0iScelu2XR5HXTAA59dw7au9uzvAymf5LmsrWIRHKc/sMvZL8aYCmJO4+0mczY6v7Nx0jO9EsNg0UzJvFk1MRIW1VV5TOfxB1pnc2M2737zp83B3GgxnQuPpOiAPUSIPFzOJoJq7umgdcz0ntFKAeM6b19ldrFnTaZu17bSJMs7uvjxUzwecO2S6G+dj8LmM+M8Za2iOcrNpovzazC~3159352~4469811; Domain=.zara.com; Path=/; Expires=Sat, 11 Mar 2023 02:48:44 GMT; Max-Age=14400
server-timing: ak_p; desc="466246_34678189_146925636_13_6784_93_0";dur=1

Maybe the problem is that urequests cannot handle cookies.

kjm1102 · 2023-03-15T06:45:43Z

kjm1102
Mar 15, 2023

urequests has had an upgrade apparently https://github.com/orgs/micropython/discussions/10788 maybe the newer version might behave more like python's requests?

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MicroPython

Micropython's urequests behaves differently than python's requests #10946

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 5 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

MicroPython

Micropython's urequests behaves differently than python's requests #10946

Uh oh!

0slb Mar 6, 2023

Replies: 2 comments · 5 replies

Uh oh!

davefes Mar 6, 2023

Uh oh!

0slb Mar 6, 2023 Author

Uh oh!

Uh oh!

Josverl Mar 6, 2023 Collaborator Sponsor

Uh oh!

0slb Mar 8, 2023 Author

Uh oh!

Josverl Mar 9, 2023 Collaborator Sponsor

Uh oh!

0slb Mar 10, 2023 Author

Uh oh!

kjm1102 Mar 15, 2023

0slb
Mar 6, 2023

Replies: 2 comments 5 replies

davefes
Mar 6, 2023

0slb Mar 6, 2023
Author

Josverl Mar 6, 2023
Collaborator Sponsor

0slb Mar 8, 2023
Author

Josverl Mar 9, 2023
Collaborator Sponsor

0slb Mar 10, 2023
Author

kjm1102
Mar 15, 2023