Fix: use new snapshot id in deleted manifest entry unless is existing entry #2266
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
geruh
reviewed
Aug 6, 2025
tests/table/test_delete.py
Outdated
|
|
||
| def assert_manifest_entry(expected_status: ManifestEntryStatus, expected_snapshot_id: int) -> None: | ||
| current_snapshot = table.refresh().current_snapshot() | ||
| manifest_files = current_snapshot.manifests(table.io) |
There was a problem hiding this comment.
Run a make lint here the snapshot calls are a union with null and require an assertion or check if they exist before calling them
There was a problem hiding this comment.
Thanks for checking! I see the CI failed with lint as well, I'll fix it in the next commit
tests/table/test_delete.py
Outdated
| pass | ||
|
|
||
|
|
||
| def test_manifest_entry_after_deletes(catalog: Catalog) -> None: |
There was a problem hiding this comment.
We can probably add this test to the existing test_deletes.py suite
Fokko
reviewed
Aug 11, 2025
There was a problem hiding this comment.
Thanks @lliangyu-lin for raising this PR, it looks indeed that we have a bug here 👍
Co-authored-by: Fokko Driesprong <[email protected]>
|
@kevinjqliu Hi Kevin, could you help re-enable the CI for this PR again? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Based on iceberg spec, when a manifest entry is marked as deleted, the snapshot id for when the entry was deleted should be used.
https://iceberg.apache.org/spec/?h=deletes#manifest-entry-fields
https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/ManifestWriter.java#L178-L179
Incorrect snapshot id could lead to data being deleted during garbage collection when not supposed to.
Are these changes tested?
Added in
test_deletes.pyAre there any user-facing changes?
No