-
Notifications
You must be signed in to change notification settings - Fork 2
Open
Labels
bugSomething isn't workingSomething isn't working
Description
Overview
I encountered an issue where reading parquet from s3 doesn't work with fsspec.
Traceback (most recent call last):
File "/opt/conda/envs/cava-metadata/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 398, in run_asgi
result = await app(self.scope, self.receive, self.send)
File "/opt/conda/envs/cava-metadata/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
return await self.app(scope, receive, send)
File "/opt/conda/envs/cava-metadata/lib/python3.8/site-packages/fastapi/applications.py", line 199, in __call__
await super().__call__(scope, receive, send)
File "/opt/conda/envs/cava-metadata/lib/python3.8/site-packages/starlette/applications.py", line 111, in __call__
await self.middleware_stack(scope, receive, send)
File "/opt/conda/envs/cava-metadata/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
raise exc from None
...
File "/opt/conda/envs/cava-metadata/lib/python3.8/site-packages/s3fs/core.py", line 993, in ls
files = maybe_sync(self._ls, self, path, refresh=refresh)
File "/opt/conda/envs/cava-metadata/lib/python3.8/site-packages/fsspec/asyn.py", line 93, in maybe_sync
return _run_until_done(func(*args, **kwargs))
File "/opt/conda/envs/cava-metadata/lib/python3.8/site-packages/fsspec/asyn.py", line 31, in _run_until_done
loop._run_once()
AttributeError: 'Loop' object has no attribute '_run_once'
This is a problem with fsspec using an internal function that only works with asyncio. The implementation of _run_until_done doesn't work with uvloop since there is no _run_once funciton in uvloop.
After scratching my head for a while, I didn't find a good solution for this issue. A temporary workaround is specifying uvicorn to use the asyncio loop rather than uvloop. This option needs to be available on the docker images https://github.com/cormorack/uvicorn-gunicorn-docker/tree/conda/docker-images
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working