Add support for Pyarrow filesystem specific properties

[geruh]

closes #2247

Rationale for this change

Add support for PyArrow filesystem specific properties allowing users to pass properties directly to the underlying PyArrow filesystem implementations (e.g. S3FileSystem, GcsFileSystem, AzureFileSystem, etc.) without requiring mapping support in PyIceberg.

Since Pyarrow is a compiled implementation, it won't soft ignore extra parameters making this functionality pretty tedious. BUt now depending on the file systems the properties can be passed through to Pyarrow and interpreted there directly.

Supported filesystems:

• https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html
• https://arrow.apache.org/docs/python/generated/pyarrow.fs.AzureFileSystem.html
• https://arrow.apache.org/docs/python/generated/pyarrow.fs.HadoopFileSystem.html
• https://arrow.apache.org/docs/python/generated/pyarrow.fs.GcsFileSystem.html
• https://arrow.apache.org/docs/python/generated/pyarrow.fs.LocalFileSystem.html

Docs:

Are these changes tested?

Added tests and tested locally with S3

Are there any user-facing changes?

Yes new capabilities, updated docs

[kevinjqliu]

Thank you for the PR!

This is a great way to unblock end users from configuring the different file systems. Instead of allowlisting each property individually, for each file system implementation constructor, this will allow us to pass args to the file system constructor.

We should verify that the implementation does not conflict with existing properties. For example, s3.access-key-id can now also be passed in as access_key in S3FileSystem but we should prioritize s3.access-key-id
https://github.com/apache/iceberg/blob/571056929091f1e62412500045f71e5ba6ea00ad/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java#L249

[kevinjqliu]

modeled after https://github.com/apache/iceberg/blob/de22fd0f998ed464251acbe31b976b4bb9669312/core/src/main/java/org/apache/iceberg/util/PropertyUtil.java#L153-L192

[kevinjqliu]

@geruh and i chatted offline about this PR and aligned on this approach.

I love that we have so many unit/integration tests, makes me feel confident about this refactor

[geruh]

I approve

[Fokko]

Sorry for jumping in so late here, I was out touching grass.

While I see what we're trying to achieve here, and provide flexibility to the user, I think it comes at a cost. First of all, the most obivous one; it creates multiple paths to set the same configuration, which might be confusing for new users. That said, the intended of allow-listing the configuration is an effort to consolidate the configuration accross implementations, but also across FileIO's. For example, s3.connect-timeout and s3.request-timeout is picked up both by fsspec and PyArrow. This avoids having to go though the documentation of each of the underlying implementation, or in the case of PyArrow, through the source code :D

I think the main question here is, to what extend we want to try to hide the implementation behind the FileIO, or do we want to expose the implementation details to the end-users. I'm leaning towards the first, but curious what others' think.