Kevinjqliu/adls #21
Kevinjqliu/adls #21
Conversation
|
/gemini review |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @kevinjqliu, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request enhances the FsspecFileIO in pyiceberg to improve how Azure Data Lake Storage (ADLS) configurations are handled. It specifically addresses an issue where ADLS account names and SAS tokens were not correctly extracted from properties when a network location was present in the URI. The changes also generalize the filesystem retrieval mechanism to pass the netloc (network location) from the URI to the underlying filesystem, allowing for more granular configuration based on the storage endpoint.
Highlights
- Improved ADLS Configuration: The _adls function now correctly parses and applies ADLS_ACCOUNT_NAME and ADLS_SAS_TOKEN from connection properties, especially when derived from the URI's network location, resolving a known issue.
- Enhanced Filesystem Property Handling: The _get_fs method in FsspecFileIO has been updated to accept and pass the netloc (network location) from the URI to the filesystem constructor, enabling more precise filesystem initialization.
- Type Hinting Refinement: The Optional type hint has been introduced for the new netloc parameter, improving code clarity and maintainability.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request refactors the ADLS filesystem initialization to correctly handle per-account SAS tokens by using the netloc from the file URI. The overall approach is sound and improves the ADLS integration. However, the changes lack unit tests to verify the new authentication logic, which is a critical gap. I've provided a comment highlighting the need for tests and another with a suggestion to simplify the implementation.
| ) | ||
|
|
||
|
|
||
| def _adls(properties: Properties) -> AbstractFileSystem: |
There was a problem hiding this comment.
This new logic for ADLS authentication is a great improvement. However, it is not covered by any unit tests. Please add tests to ensure this works as expected and to prevent future regressions. A unit test could mock adlfs.AzureBlobFileSystem and verify it's called with the correct account_name and sas_token when per-account credentials are provided in the properties.
| if netloc := properties.get("netloc"): | ||
| account_uri = netloc.split("@")[-1] | ||
| else: | ||
| account_uri = None | ||
|
|
||
| if not properties.get(ADLS_ACCOUNT_NAME) and account_uri: | ||
| properties[ADLS_ACCOUNT_NAME] = account_uri.split(".")[0] | ||
|
|
||
| # Fixes https://github.com/apache/iceberg-python/issues/1146 | ||
| if not properties.get(ADLS_SAS_TOKEN) and account_uri: | ||
| properties[ADLS_SAS_TOKEN] = properties.get(f"{ADLS_SAS_TOKEN}.{account_uri}") |
There was a problem hiding this comment.
This logic can be slightly simplified by nesting the checks inside the if netloc := ... block. This avoids the need for the else clause and checking for account_uri being non-None in subsequent if statements, making the code more concise and readable.
| if netloc := properties.get("netloc"): | |
| account_uri = netloc.split("@")[-1] | |
| else: | |
| account_uri = None | |
| if not properties.get(ADLS_ACCOUNT_NAME) and account_uri: | |
| properties[ADLS_ACCOUNT_NAME] = account_uri.split(".")[0] | |
| # Fixes https://github.com/apache/iceberg-python/issues/1146 | |
| if not properties.get(ADLS_SAS_TOKEN) and account_uri: | |
| properties[ADLS_SAS_TOKEN] = properties.get(f"{ADLS_SAS_TOKEN}.{account_uri}") | |
| if netloc := properties.get("netloc"): | |
| account_uri = netloc.split("@")[-1] | |
| if not properties.get(ADLS_ACCOUNT_NAME): | |
| properties[ADLS_ACCOUNT_NAME] = account_uri.split(".")[0] | |
| # Fixes https://github.com/apache/iceberg-python/issues/1146 | |
| if not properties.get(ADLS_SAS_TOKEN): | |
| properties[ADLS_SAS_TOKEN] = properties.get(f"{ADLS_SAS_TOKEN}.{account_uri}") |
There was a problem hiding this comment.
Code Review
This pull request introduces significant improvements to the Azure Data Lake Storage (ADLS) integration by correctly handling per-account credentials, specifically SAS tokens. The changes involve passing the netloc from file URIs down to the filesystem factory, allowing for proper credential resolution for the specific account being accessed. A key enhancement is the copying of properties before they are used to initialize a filesystem, which prevents state leakage and resolves a potential bug. Overall, these changes make the ADLS support more robust and correct. I have one suggestion to slightly refactor a section of the new logic in the _adls function to improve its clarity and maintainability.
| if netloc := properties.get("netloc"): | ||
| account_uri = netloc.split("@")[-1] | ||
| else: | ||
| account_uri = None | ||
|
|
||
| if not properties.get(ADLS_ACCOUNT_NAME) and account_uri: | ||
| properties[ADLS_ACCOUNT_NAME] = account_uri.split(".")[0] | ||
|
|
||
| # Fixes https://github.com/apache/iceberg-python/issues/1146 | ||
| if not properties.get(ADLS_SAS_TOKEN) and account_uri: | ||
| properties[ADLS_SAS_TOKEN] = properties.get(f"{ADLS_SAS_TOKEN}.{account_uri}") |
There was a problem hiding this comment.
This block of code can be refactored for better readability. By first determining account_uri and then using a single conditional block, you can avoid the else statement and repeated and account_uri checks. This streamlines the logic and makes it easier to follow.
| if netloc := properties.get("netloc"): | |
| account_uri = netloc.split("@")[-1] | |
| else: | |
| account_uri = None | |
| if not properties.get(ADLS_ACCOUNT_NAME) and account_uri: | |
| properties[ADLS_ACCOUNT_NAME] = account_uri.split(".")[0] | |
| # Fixes https://github.com/apache/iceberg-python/issues/1146 | |
| if not properties.get(ADLS_SAS_TOKEN) and account_uri: | |
| properties[ADLS_SAS_TOKEN] = properties.get(f"{ADLS_SAS_TOKEN}.{account_uri}") | |
| account_uri = None | |
| if netloc := properties.get("netloc"): | |
| account_uri = netloc.split("@")[-1] | |
| if account_uri: | |
| if not properties.get(ADLS_ACCOUNT_NAME): | |
| properties[ADLS_ACCOUNT_NAME] = account_uri.split(".")[0] | |
| # Fixes https://github.com/apache/iceberg-python/issues/1146 | |
| if not properties.get(ADLS_SAS_TOKEN): | |
| properties[ADLS_SAS_TOKEN] = properties.get(f"{ADLS_SAS_TOKEN}.{account_uri}") |
Rationale for this change
Are these changes tested?
Are there any user-facing changes?