You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
| pyarrow.use-large-types-on-read | True | Use large PyArrow types i.e. [large_string](https://arrow.apache.org/docs/python/generated/pyarrow.large_string.html), [large_binary](https://arrow.apache.org/docs/python/generated/pyarrow.large_binary.html) and [large_list](https://arrow.apache.org/docs/python/generated/pyarrow.large_list.html) field types on table scans. The default value is True. |
223
221
224
-
<!-- markdown-link-check-enable-->
225
222
#### Advanced FileSystem Configuration
226
223
227
224
When using `PyArrowFileIO`, you can **pass additional configuration properties directly to the underlying PyArrow filesystem implementations**. This feature enables you to use any PyArrow filesystem option without requiring explicit PyIceberg support.
@@ -248,8 +245,6 @@ Where:
248
245
249
246
##### Supported Prefixes and FileSystems
250
247
251
-
<!-- markdown-link-check-disable -->
252
-
253
248
| Property Prefix | FileSystem | Example | Description |
|`s3.`|[S3FileSystem](https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html)|`s3.load_frequency=900`| Passed as `load_frequency=900` to S3FileSystem |
@@ -259,8 +254,6 @@ Where:
259
254
|`oss.`|[S3FileSystem](https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html)|`oss.connect_timeout=30.0`| Passed as `connect_timeout=30.0` to S3FileSystem |
260
255
|`file.`|[LocalFileSystem](https://arrow.apache.org/docs/python/generated/pyarrow.fs.LocalFileSystem.html)|`file.use_mmap=true`| Passed as `use_mmap=True` to LocalFileSystem |
261
256
262
-
<!-- markdown-link-check-enable -->
263
-
264
257
**Note:** Refer to the PyArrow documentation for each filesystem to understand the available parameters and their expected types. Property values are passed directly to PyArrow, so they must match the exact parameter names and types expected by the filesystem constructors.
0 commit comments