Skip to content

Commit 3457bc2

Browse files
use PyArrowFileIO as default for abfs and wasb schemes (#2395)
<!-- Thanks for opening a pull request! --> <!-- In the case this PR will resolve an issue, please replace ${GITHUB_ISSUE_ID} below with the actual Github issue id. --> <!-- Closes #${GITHUB_ISSUE_ID} --> # Rationale for this change Now that [pyarrow FileIO supports ADLS](#2111), we can update the [SCHEMA_TO_FILE_IO](https://github.com/apache/iceberg-python/blob/370e770383c9c129aff2fe24cc92a68981688159/pyiceberg/io/__init__.py#L307-L319) mapping for `abfs` and `wasb` to use ARROW_FILE_IO, similar to how it’s handled for s3 We’re keeping FsspecFileIO as the preferred default, since the PyArrowFileIO implementation is only available in pyarrow >= 20.0.0 ## Are these changes tested? ## Are there any user-facing changes? Add mapping SCHEMA_TO_FILE_IO for `wasb` and `abfs` to `FsspecFileIO` and `PyArrowFileIO` <!-- In the case of user-facing changes, please add the changelog label. -->
1 parent 3eecdad commit 3457bc2

File tree

1 file changed

+4
-2
lines changed

1 file changed

+4
-2
lines changed

pyiceberg/io/__init__.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -313,8 +313,10 @@ def delete(self, location: Union[str, InputFile, OutputFile]) -> None:
313313
"file": [ARROW_FILE_IO, FSSPEC_FILE_IO],
314314
"hdfs": [ARROW_FILE_IO],
315315
"viewfs": [ARROW_FILE_IO],
316-
"abfs": [FSSPEC_FILE_IO],
317-
"abfss": [FSSPEC_FILE_IO],
316+
"abfs": [FSSPEC_FILE_IO, ARROW_FILE_IO],
317+
"abfss": [FSSPEC_FILE_IO, ARROW_FILE_IO],
318+
"wasb": [FSSPEC_FILE_IO, ARROW_FILE_IO],
319+
"wasbs": [FSSPEC_FILE_IO, ARROW_FILE_IO],
318320
"hf": [FSSPEC_FILE_IO],
319321
}
320322

0 commit comments

Comments
 (0)