Skip to content

Commit b99a8aa

Browse files
veitveit
committed
📝 Switch from the requests lib to httpx
The requests lib feels dead. See also https://blog.ian.stapletoncordas.co/2024/02/a-retrospective-on-requests
1 parent 5e50ecb commit b99a8aa

File tree

11 files changed

+119
-187
lines changed

11 files changed

+119
-187
lines changed

‎docs/data-processing/apis/fastapi/install.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ Pydantic can use the optional dependencies
4646

4747
Starlette can use the optional dependencies
4848

49-
:doc:`requests <../../requests/index>`
49+
:doc:`httpx <../../httpx/index>`
5050
if you want to use the ``TestClient``.
5151
`aiofiles <https://github.com/Tinche/aiofiles>`_
5252
if you want to use ``FileResponse`` or ``StaticFiles``.

docs/data-processing/requests/module.ipynb.license renamed to docs/data-processing/httpx/httpx.ipynb.license

File renamed without changes.

‎docs/data-processing/httpx/index.rst‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
.. SPDX-FileCopyrightText: 2021 Veit Schiele
2+
..
3+
.. SPDX-License-Identifier: BSD-3-Clause
4+
5+
httpx
6+
=====
7+
8+
`httpx <https://www.python-httpx.org>`_ is an http client with which requests
9+
can be sent easily.
10+
11+
.. toctree::
12+
:hidden:
13+
:titlesonly:
14+
:maxdepth: 0
15+
16+
install-example.ipynb
17+
module.ipynb

docs/data-processing/requests/requests.ipynb renamed to docs/data-processing/httpx/install-example.ipynb

Lines changed: 85 additions & 120 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
"cell_type": "markdown",
55
"metadata": {},
66
"source": [
7-
"# Requests installation and sample application"
7+
"# httpx installation and sample application"
88
]
99
},
1010
{
@@ -13,17 +13,17 @@
1313
"source": [
1414
"## Installation\n",
1515
"\n",
16-
"The requests library is useful for communicating with REST APIs. With [Spack](../../productive/envs/spack/index.rst) you can provide requests in your kernel:\n",
16+
"The httpx library is useful for communicating with REST APIs. With [Spack](../../productive/envs/spack/index.rst) you can provide httpx in your kernel:\n",
1717
"\n",
1818
"``` bash\n",
1919
"$ spack env activate python-311\n",
20-
"$ spack install py-requests\n",
20+
"$ spack install py-httpx\n",
2121
"```\n",
2222
"\n",
23-
"Alternatively, you can install requests with other package managers, for example\n",
23+
"Alternatively, you can install httpx with other package managers, for example\n",
2424
"\n",
2525
"``` bash\n",
26-
"$ pipenv install requests\n",
26+
"$ pipenv install httpx\n",
2727
"```"
2828
]
2929
},
@@ -35,7 +35,7 @@
3535
"\n",
3636
"In this example we get our data from the [OpenStreetMap Nominatim API](https://nominatim.org/release-docs/develop/api/Overview/#nominatim-api). This can be reached via the URL `https://nominatim.openstreetmap.org/search?`. To e.g. receive information about the Berlin Congress Center in Berlin in JSON format, the URL `https://nominatim.openstreetmap.org/search.php?q=Alexanderplatz+Berlin&format=json` should be given, and if you want to display the corresponding map section you just have to leave out `&format=json`.\n",
3737
"\n",
38-
"Then we define the base URL and the parameters. Nominatim expects at least the following two parameters\n",
38+
"Then we define the search URL and the parameters. Nominatim expects at least the following two parameters\n",
3939
"\n",
4040
"| Key | Value |\n",
4141
"| --------- | ------------------------------------ |\n",
@@ -51,15 +51,15 @@
5151
"metadata": {},
5252
"outputs": [],
5353
"source": [
54-
"import requests\n",
54+
"import httpx\n",
5555
"\n",
5656
"\n",
57-
"base_url = \"https://nominatim.openstreetmap.org/search?\"\n",
57+
"search_url = \"https://nominatim.openstreetmap.org/search?\"\n",
5858
"params = {\n",
5959
" \"q\": \"Alexanderplatz, Berlin\",\n",
6060
" \"format\": \"json\",\n",
6161
"}\n",
62-
"r = requests.get(base_url, params=params)"
62+
"r = httpx.get(search_url, params=params)"
6363
]
6464
},
6565
{
@@ -92,75 +92,62 @@
9292
{
9393
"data": {
9494
"text/plain": [
95-
"[{'place_id': 261767431,\n",
96-
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',\n",
95+
"[{'place_id': 128497332,\n",
96+
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright',\n",
9797
" 'osm_type': 'way',\n",
9898
" 'osm_id': 783052052,\n",
99-
" 'boundingbox': ['52.5201457', '52.5238113', '13.4103097', '13.4160801'],\n",
10099
" 'lat': '52.5219814',\n",
101100
" 'lon': '13.413635717448294',\n",
102-
" 'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',\n",
103101
" 'class': 'place',\n",
104102
" 'type': 'square',\n",
105-
" 'importance': 0.6914982526373583},\n",
106-
" {'place_id': 45802928,\n",
107-
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',\n",
103+
" 'place_rank': 25,\n",
104+
" 'importance': 0.47149825263735834,\n",
105+
" 'addresstype': 'square',\n",
106+
" 'name': 'Alexanderplatz',\n",
107+
" 'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',\n",
108+
" 'boundingbox': ['52.5201457', '52.5238113', '13.4103097', '13.4160801']},\n",
109+
" {'place_id': 128243381,\n",
110+
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright',\n",
108111
" 'osm_type': 'node',\n",
109112
" 'osm_id': 3908141014,\n",
110-
" 'boundingbox': ['52.5165661', '52.5265661', '13.4062804', '13.4162804'],\n",
111113
" 'lat': '52.5215661',\n",
112114
" 'lon': '13.4112804',\n",
113-
" 'display_name': 'Alexanderplatz, Dircksenstraße, Mitte, Berlin, 10179, Deutschland',\n",
114115
" 'class': 'railway',\n",
115116
" 'type': 'station',\n",
116-
" 'importance': 0.6560990777880803,\n",
117-
" 'icon': 'https://nominatim.openstreetmap.org/ui/mapicons/transport_train_station2.p.20.png'},\n",
118-
" {'place_id': 190976103,\n",
119-
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',\n",
117+
" 'place_rank': 30,\n",
118+
" 'importance': 0.43609907778808027,\n",
119+
" 'addresstype': 'railway',\n",
120+
" 'name': 'Alexanderplatz',\n",
121+
" 'display_name': 'Alexanderplatz, Dircksenstraße, Mitte, Berlin, 10179, Deutschland',\n",
122+
" 'boundingbox': ['52.5165661', '52.5265661', '13.4062804', '13.4162804']},\n",
123+
" {'place_id': 128416772,\n",
124+
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright',\n",
120125
" 'osm_type': 'way',\n",
121126
" 'osm_id': 346206374,\n",
122-
" 'boundingbox': ['52.5216214', '52.5216661', '13.4131913', '13.4131914'],\n",
123127
" 'lat': '52.5216214',\n",
124128
" 'lon': '13.4131913',\n",
125-
" 'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',\n",
126129
" 'class': 'highway',\n",
127130
" 'type': 'pedestrian',\n",
128-
" 'importance': 0.32000999999999996},\n",
129-
" {'place_id': 51760167,\n",
130-
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',\n",
131-
" 'osm_type': 'node',\n",
132-
" 'osm_id': 4598414751,\n",
133-
" 'boundingbox': ['52.5165849', '52.5265849', '13.4089082', '13.4189082'],\n",
134-
" 'lat': '52.5215849',\n",
135-
" 'lon': '13.4139082',\n",
131+
" 'place_rank': 26,\n",
132+
" 'importance': 0.10000999999999993,\n",
133+
" 'addresstype': 'road',\n",
134+
" 'name': 'Alexanderplatz',\n",
136135
" 'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',\n",
137-
" 'class': 'railway',\n",
138-
" 'type': 'station',\n",
139-
" 'importance': 0.22000999999999998,\n",
140-
" 'icon': 'https://nominatim.openstreetmap.org/ui/mapicons/transport_train_station2.p.20.png'},\n",
141-
" {'place_id': 49544606,\n",
142-
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',\n",
143-
" 'osm_type': 'node',\n",
144-
" 'osm_id': 4389211800,\n",
145-
" 'boundingbox': ['52.5231653', '52.5232653', '13.414475', '13.414575'],\n",
146-
" 'lat': '52.5232153',\n",
147-
" 'lon': '13.414525',\n",
148-
" 'display_name': 'Alexanderplatz, Alexanderstraße, Mitte, Berlin, 10178, Deutschland',\n",
149-
" 'class': 'highway',\n",
150-
" 'type': 'bus_stop',\n",
151-
" 'importance': 0.22000999999999998,\n",
152-
" 'icon': 'https://nominatim.openstreetmap.org/ui/mapicons/transport_bus_stop2.p.20.png'},\n",
153-
" {'place_id': 181190866,\n",
154-
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',\n",
136+
" 'boundingbox': ['52.5216214', '52.5216661', '13.4131913', '13.4131914']},\n",
137+
" {'place_id': 127680907,\n",
138+
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright',\n",
155139
" 'osm_type': 'way',\n",
156-
" 'osm_id': 301241483,\n",
157-
" 'boundingbox': ['52.5226699', '52.5227698', '13.4152008', '13.4154146'],\n",
158-
" 'lat': '52.5227318',\n",
159-
" 'lon': '13.4152821',\n",
160-
" 'display_name': 'Alexanderstraße, Mitte, Berlin, 10178, Deutschland',\n",
140+
" 'osm_id': 301733776,\n",
141+
" 'lat': '52.5222454',\n",
142+
" 'lon': '13.4158136',\n",
161143
" 'class': 'highway',\n",
162144
" 'type': 'primary',\n",
163-
" 'importance': 0.21000999999999995}]"
145+
" 'place_rank': 26,\n",
146+
" 'importance': 0.10000999999999993,\n",
147+
" 'addresstype': 'road',\n",
148+
" 'name': 'Alexanderstraße',\n",
149+
" 'display_name': 'Alexanderstraße, Mitte, Berlin, 10178, Deutschland',\n",
150+
" 'boundingbox': ['52.5222454', '52.5224356', '13.4153983', '13.4158136']}]"
164151
]
165152
},
166153
"execution_count": 3,
@@ -187,17 +174,20 @@
187174
{
188175
"data": {
189176
"text/plain": [
190-
"[{'place_id': 261767431,\n",
191-
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',\n",
177+
"[{'place_id': 128497332,\n",
178+
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright',\n",
192179
" 'osm_type': 'way',\n",
193180
" 'osm_id': 783052052,\n",
194-
" 'boundingbox': ['52.5201457', '52.5238113', '13.4103097', '13.4160801'],\n",
195181
" 'lat': '52.5219814',\n",
196182
" 'lon': '13.413635717448294',\n",
197-
" 'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',\n",
198183
" 'class': 'place',\n",
199184
" 'type': 'square',\n",
200-
" 'importance': 0.6914982526373583}]"
185+
" 'place_rank': 25,\n",
186+
" 'importance': 0.47149825263735834,\n",
187+
" 'addresstype': 'square',\n",
188+
" 'name': 'Alexanderplatz',\n",
189+
" 'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',\n",
190+
" 'boundingbox': ['52.5201457', '52.5238113', '13.4103097', '13.4160801']}]"
201191
]
202192
},
203193
"execution_count": 4,
@@ -207,7 +197,7 @@
207197
],
208198
"source": [
209199
"params = {\"q\": \"Alexanderplatz, Berlin\", \"format\": \"json\", \"limit\": \"1\"}\n",
210-
"r = requests.get(base_url, params=params)\n",
200+
"r = httpx.get(search_url, params=params)\n",
211201
"r.json()"
212202
]
213203
},
@@ -229,14 +219,25 @@
229219
"cell_type": "markdown",
230220
"metadata": {},
231221
"source": [
232-
"To ensure that the interaction was successful, we use the `raise_for_status` method of `requests`, which throws an exception if the HTTP status code isn’t `200 OK`:"
222+
"To ensure that the interaction was successful, we use the `raise_for_status` method of `httpx`, which throws an exception if the HTTP status code isn’t `200 OK`:"
233223
]
234224
},
235225
{
236226
"cell_type": "code",
237227
"execution_count": 5,
238228
"metadata": {},
239-
"outputs": [],
229+
"outputs": [
230+
{
231+
"data": {
232+
"text/plain": [
233+
"<Response [200 OK]>"
234+
]
235+
},
236+
"execution_count": 5,
237+
"metadata": {},
238+
"output_type": "execute_result"
239+
}
240+
],
240241
"source": [
241242
"r.raise_for_status()"
242243
]
@@ -245,7 +246,7 @@
245246
"cell_type": "markdown",
246247
"metadata": {},
247248
"source": [
248-
"Since we don't want to exceed the load limits of the Nominatim API, we will delay our requests with the `time.sleep` function:"
249+
"Since we don't want to exceed the load limits of the Nominatim API, we will delay our httpx with the `time.sleep` function:"
249250
]
250251
},
251252
{
@@ -256,17 +257,20 @@
256257
{
257258
"data": {
258259
"text/plain": [
259-
"[{'place_id': 261767431,\n",
260-
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',\n",
260+
"[{'place_id': 128497332,\n",
261+
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright',\n",
261262
" 'osm_type': 'way',\n",
262263
" 'osm_id': 783052052,\n",
263-
" 'boundingbox': ['52.5201457', '52.5238113', '13.4103097', '13.4160801'],\n",
264264
" 'lat': '52.5219814',\n",
265265
" 'lon': '13.413635717448294',\n",
266-
" 'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',\n",
267266
" 'class': 'place',\n",
268267
" 'type': 'square',\n",
269-
" 'importance': 0.6914982526373583}]"
268+
" 'place_rank': 25,\n",
269+
" 'importance': 0.47149825263735834,\n",
270+
" 'addresstype': 'square',\n",
271+
" 'name': 'Alexanderplatz',\n",
272+
" 'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',\n",
273+
" 'boundingbox': ['52.5201457', '52.5238113', '13.4103097', '13.4160801']}]"
270274
]
271275
},
272276
"execution_count": 6,
@@ -302,7 +306,7 @@
302306
" \"\"\"\n",
303307
" search_url = \"https://nominatim.openstreetmap.org/search?\"\n",
304308
" params = {\"q\": address, \"format\": format, \"limit\": limit, **kwargs}\n",
305-
" r = requests.get(search_url, params=params)\n",
309+
" r = httpx.get(search_url, params=params)\n",
306310
" # Raise an exception if the status is unsuccessful\n",
307311
" r.raise_for_status()\n",
308312
"\n",
@@ -325,17 +329,20 @@
325329
{
326330
"data": {
327331
"text/plain": [
328-
"[{'place_id': 261767431,\n",
329-
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',\n",
332+
"[{'place_id': 128497332,\n",
333+
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright',\n",
330334
" 'osm_type': 'way',\n",
331335
" 'osm_id': 783052052,\n",
332-
" 'boundingbox': ['52.5201457', '52.5238113', '13.4103097', '13.4160801'],\n",
333336
" 'lat': '52.5219814',\n",
334337
" 'lon': '13.413635717448294',\n",
335-
" 'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',\n",
336338
" 'class': 'place',\n",
337339
" 'type': 'square',\n",
338-
" 'importance': 0.6914982526373583}]"
340+
" 'place_rank': 25,\n",
341+
" 'importance': 0.47149825263735834,\n",
342+
" 'addresstype': 'square',\n",
343+
" 'name': 'Alexanderplatz',\n",
344+
" 'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',\n",
345+
" 'boundingbox': ['52.5201457', '52.5238113', '13.4103097', '13.4160801']}]"
339346
]
340347
},
341348
"execution_count": 8,
@@ -347,48 +354,6 @@
347354
"nominatim_search(\"Alexanderplatz, Berlin\")"
348355
]
349356
},
350-
{
351-
"cell_type": "markdown",
352-
"metadata": {},
353-
"source": [
354-
"However, you can use other parameters besides `address`. You can get an overview in the [Nominatim Docs](https://nominatim.org/release-docs/develop/api/Search/#parameters)."
355-
]
356-
},
357-
{
358-
"cell_type": "code",
359-
"execution_count": 9,
360-
"metadata": {},
361-
"outputs": [
362-
{
363-
"data": {
364-
"text/plain": [
365-
"[{'place_id': 194632110,\n",
366-
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',\n",
367-
" 'osm_type': 'way',\n",
368-
" 'osm_id': 368946992,\n",
369-
" 'boundingbox': ['52.5341829', '52.534317', '13.4237697', '13.4240606'],\n",
370-
" 'lat': '52.53425085',\n",
371-
" 'lon': '13.423921421750569',\n",
372-
" 'display_name': '8, Marienburger Straße, Winsviertel, Prenzlauer Berg, Pankow, Berlin, 10405, Deutschland',\n",
373-
" 'class': 'building',\n",
374-
" 'type': 'apartments',\n",
375-
" 'importance': 0.4200099999999999}]"
376-
]
377-
},
378-
"execution_count": 9,
379-
"metadata": {},
380-
"output_type": "execute_result"
381-
}
382-
],
383-
"source": [
384-
"nominatim_search(\n",
385-
" address=None,\n",
386-
" street=\"8, Marienburger Straße\",\n",
387-
" city=\"Berlin\",\n",
388-
" country=\"Germany\",\n",
389-
")"
390-
]
391-
},
392357
{
393358
"cell_type": "markdown",
394359
"metadata": {},
@@ -400,7 +365,7 @@
400365
},
401366
{
402367
"cell_type": "code",
403-
"execution_count": 10,
368+
"execution_count": 9,
404369
"metadata": {},
405370
"outputs": [],
406371
"source": [
@@ -436,7 +401,7 @@
436401
"name": "python",
437402
"nbconvert_exporter": "python",
438403
"pygments_lexer": "ipython3",
439-
"version": "3.11.4"
404+
"version": "3.11.5"
440405
},
441406
"latex_envs": {
442407
"LaTeX_envs_menu_present": true,

0 commit comments

Comments
 (0)