Fix filesystem

[kevinjqliu]

Rationale for this change

Are these changes tested?

Are there any user-facing changes?

[gemini-code-assist]

Summary of Changes

Hello @kevinjqliu, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refines how filesystem locations are parsed within pyiceberg by introducing configurable default schemes and network locations. This change enhances the flexibility and robustness of file I/O operations, allowing for more dynamic and environment-specific handling of paths that lack explicit URI schemes.

Highlights

• Enhanced Filesystem Location Parsing: The parse_location static method in pyiceberg/io/pyarrow.py now accepts an optional properties dictionary. This allows for configurable default schemes and network locations (e.g., DEFAULT_SCHEME, DEFAULT_NETLOC) to be used when a URI scheme is not explicitly provided in the location string, improving flexibility.
• Integration of New Parsing Logic: The new_input, new_output, and delete methods within PyArrowFileIO now pass the instance's properties to the updated parse_location method. This ensures that the configurable default scheme and network location are consistently applied across various file operations.
• New Test Coverage: A new test case, test_parse_location_defaults, has been added to tests/io/test_pyarrow.py. This test validates the correct application of default schemes and network locations by the parse_location method, ensuring the new functionality works as expected.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request enhances PyArrowFileIO by allowing the default scheme and netloc for schemeless paths to be configured through properties. This is a good improvement for flexibility. The changes are applied consistently across new_input, new_output, and delete methods. The new tests correctly cover the new functionality. I've provided one suggestion to make the new test more robust and platform-independent.

[gemini-code-assist]

The test assertions for path are not platform-independent. The parse_location function uses os.path.abspath(location) for schemeless paths, but the test hardcodes the expected path as "/foo/bar". This will work on Unix-like systems but fail on Windows where the absolute path would be different (e.g., C:\\foo\\bar).

To make the test robust across different operating systems, you should compare the result with os.path.abspath().

Additionally, the import from pyiceberg.io.pyarrow import PyArrowFileIO is inside the test function. It's better to move it to the top of the file to follow the common style in this project. I have removed it in the suggestion below.

def test_parse_location_defaults() -> None:
    """Test that parse_location uses defaults."""

    location = "/foo/bar"
    expected_path = os.path.abspath(location)

    # if no default scheme or netloc is provided, use file scheme and empty netloc
    scheme, netloc, path = PyArrowFileIO.parse_location(location)
    assert scheme == "file"
    assert netloc == ""
    assert path == expected_path

    scheme, netloc, path = PyArrowFileIO.parse_location(
        location, properties={"DEFAULT_SCHEME": "scheme", "DEFAULT_NETLOC": "netloc:8000"}
    )
    assert scheme == "scheme"
    assert netloc == "netloc:8000"
    assert path == expected_path

    scheme, netloc, path = PyArrowFileIO.parse_location(
        location, properties={"DEFAULT_SCHEME": "hdfs", "DEFAULT_NETLOC": "netloc:8000"}
    )
    assert scheme == "hdfs"
    assert netloc == "netloc:8000"
    assert path == expected_path