Skip to content

Can't read parquet with uvloop loop option #13

@lsetiawan

Description

@lsetiawan

Overview

I encountered an issue where reading parquet from s3 doesn't work with fsspec.

Traceback (most recent call last):
  File "/opt/conda/envs/cava-metadata/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 398, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/opt/conda/envs/cava-metadata/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/opt/conda/envs/cava-metadata/lib/python3.8/site-packages/fastapi/applications.py", line 199, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/conda/envs/cava-metadata/lib/python3.8/site-packages/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/conda/envs/cava-metadata/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
...
  File "/opt/conda/envs/cava-metadata/lib/python3.8/site-packages/s3fs/core.py", line 993, in ls
    files = maybe_sync(self._ls, self, path, refresh=refresh)
  File "/opt/conda/envs/cava-metadata/lib/python3.8/site-packages/fsspec/asyn.py", line 93, in maybe_sync
    return _run_until_done(func(*args, **kwargs))
  File "/opt/conda/envs/cava-metadata/lib/python3.8/site-packages/fsspec/asyn.py", line 31, in _run_until_done
    loop._run_once()
AttributeError: 'Loop' object has no attribute '_run_once'

This is a problem with fsspec using an internal function that only works with asyncio. The implementation of _run_until_done doesn't work with uvloop since there is no _run_once funciton in uvloop.

After scratching my head for a while, I didn't find a good solution for this issue. A temporary workaround is specifying uvicorn to use the asyncio loop rather than uvloop. This option needs to be available on the docker images https://github.com/cormorack/uvicorn-gunicorn-docker/tree/conda/docker-images

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions