-
Notifications
You must be signed in to change notification settings - Fork 408
Open
Description
Hey!
ArrowFSWrapper
has been very helpful to me. I noticed, however, that it cannot do ls
on a single object properly. Here's what I'm doing:
In [1]: !aws s3 ls s3://bucket/foo.txt
2021-11-25 13:24:11 40299 foo.txt
from pyarrow.fs import S3FileSystem
from fsspec.implementations.arrow import ArrowFSWrapper
fs = ArrowFSWrapper(S3FileSystem())
fs.ls("s3://bucket/foo.txt")
the result is
FileNotFoundError Traceback (most recent call last)
Cell In[5], line 1
----> 1 fs.ls("s3://bucket/foo.txt")
File ~/.venv/lib/python3.10/site-packages/fsspec/implementations/arrow.py:80, in ArrowFSWrapper.ls(self, path, detail, **kwargs)
75 path = self._strip_protocol(path)
76 from pyarrow.fs import FileSelector
78 entries = [
79 self._make_entry(entry)
---> 80 for entry in self.fs.get_file_info(FileSelector(path))
81 ]
82 if detail:
83 return entries
File ~/.venv/lib/python3.10/site-packages/pyarrow/_fs.pyx:607, in pyarrow._fs.FileSystem.get_file_info()
File ~/.venv/lib/python3.10/site-packages/pyarrow/error.pxi:155, in pyarrow.lib.pyarrow_internal_check_status()
File ~/.venv/lib/python3.10/site-packages/pyarrow/error.pxi:92, in pyarrow.lib.check_status()
FileNotFoundError: [Errno 2] Path does not exist 'bucket/foo.txt/'. Detail: [errno 2] No such file or directory
It seems that ArrowFSWrapper.ls
adds a trailing slash /
to the name. Then, it doesn't exist, it's true.
It looks as if ls
was written with folders in mind, primarily. However, ls
on a single object should work too, I suppose.
Is this the case or I misunderstood something?
Metadata
Metadata
Assignees
Labels
No labels