Skip to content

Commit 59d8dba

Browse files
committed
ReferenceFileSystem: use fs.open instead of fs._open
There is a bug in `fsspec==2024.12.0` that causes the `ReferenceFileSystem` to incorrectly make `fs._open` return a coroutine object instead of a file-like object. (See a proposed PR to fix this issue: fsspec/filesystem_spec#1769.) We have a test for the expected behavior (`test_arrow_generator_partitioned` in `tests/unit/lib/test_arrow.py`) running in the CI environment. But that does not fail because the latest version of `fsspec` does not get installed in the CI due to the upper limit set by the `datasets` library. The `datasets` library is only installed as part of the `hf` and `tests` extras, so the default installation of `datachain` will encounter this issue. Fixes #806.
1 parent aad99e2 commit 59d8dba

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

src/datachain/lib/arrow.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ def _open(self, path, mode="rb", *args, **kwargs):
3333
# reads the whole file in-memory.
3434
(uri,) = self.references[path]
3535
protocol, _ = split_protocol(uri)
36-
return self.fss[protocol]._open(uri, mode, *args, **kwargs)
36+
return self.fss[protocol].open(uri, mode, *args, **kwargs)
3737

3838

3939
class ArrowGenerator(Generator):

0 commit comments

Comments
 (0)