-
Notifications
You must be signed in to change notification settings - Fork 2.5k
fix: Handle external file groups in ExternalFilePathUtil #17788
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
nsivabalan
merged 4 commits into
apache:master
from
vinishjail97:Hudi-ExternalFileGroup
Jan 26, 2026
Merged
fix: Handle external file groups in ExternalFilePathUtil #17788
nsivabalan
merged 4 commits into
apache:master
from
vinishjail97:Hudi-ExternalFileGroup
Jan 26, 2026
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4 tasks
Contributor
Author
There are temporary dependency resolution errors from maven. |
e4f8546 to
9d12f46
Compare
2 tasks
vinishjail97
commented
Jan 16, 2026
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java
Show resolved
Hide resolved
9328ff3 to
0682a50
Compare
hudi-common/src/main/java/org/apache/hudi/common/util/ExternalFilePathUtil.java
Outdated
Show resolved
Hide resolved
hudi-common/src/main/java/org/apache/hudi/common/util/ExternalFilePathUtil.java
Outdated
Show resolved
Hide resolved
hudi-common/src/main/java/org/apache/hudi/common/util/ExternalFilePathUtil.java
Outdated
Show resolved
Hide resolved
the-other-tim-brown
approved these changes
Jan 23, 2026
Collaborator
nsivabalan
reviewed
Jan 24, 2026
hudi-common/src/main/java/org/apache/hudi/common/util/ExternalFilePathUtil.java
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
PR adds support for external base files with file group prefixes, allowing external files to be organized in subdirectories within partitions. (Paimon for example organizes parquet files into partitions and each partition has multiple buckets).
The implementation consolidates all external file handling logic in ExternalFilePathUtil for better maintainability.
Summary and Changelog
Support for external files with file group prefix format:
<fileName>_<commitTime>_fg%3D<prefix>_hudiextConsolidated external file parsing logic in
ExternalFilePathUtil.parseFileIdAndCommitTimeFromExternalFile()Added comprehensive test suite for ExternalFilePathUtil.
Impact
Enables hudi metadata generation for external files that are organized in subdirectories within partitions, useful for generating hudi metadata for Paimon tables.
Risk Level
Low. The changes maintain backward compatibility with existing external files (without prefix) while adding support for the new format.
Documentation Update
None. Updated java doc for ExternalFilePathUtil.
Contributor's checklist