httpx.AsyncClient not able to resolve hostname after forking new process #2756
-
|
Hi, I used httpx to write a connector library for some API and it works great. Now, when using it in an Airflow DAG, I noticed that making requests using Nevermind the Airflow jargon, though. I have reduced the problem to the following minimum failing example:
import asyncio
from datetime import datetime
import logging
import os
import traceback
import httpx
def download_and_log(f):
async def httpx_wrapper():
async with httpx.AsyncClient() as cli:
try:
resp = await cli.get("https://example.com/")
print(f"Successful request: {resp.text[:100]}", file=f)
except:
traceback.print_exc(file=f)
asyncio.get_event_loop().run_until_complete(httpx_wrapper())
with open("/tmp/test_httpx_parent", "w") as f:
download_and_log(f)
pid = os.fork()
if pid:
pid, ret = os.waitpid(pid, 0)
else:
with open("/tmp/test_httpx_child", "w") as f:
download_and_log(f)Here's the full traceback from I suspect that I used |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 3 replies
-
|
You're using threads and asyncio primitives in a way that I don't really know how they'll interact. Before you try anything else, switch this...
To this...
Then let us know what you're seeing. |
Beta Was this translation helpful? Give feedback.
-
|
Using That's probably enough for me to solve what I'm doing, but do you think the behavior with |
Beta Was this translation helpful? Give feedback.
-
No idea at all. Python's If I'm working on async code I'd always prefer using |
Beta Was this translation helpful? Give feedback.
-
|
Alright, makes sense. I'll mark this as answered then. Thanks for the quick help! |
Beta Was this translation helpful? Give feedback.
You're using threads and asyncio primitives in a way that I don't really know how they'll interact.
Before you try anything else, switch this...
To this...
Then let us know what you're seeing.