You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
fix(reader): Support both position and equality delete files on the same FileScanTask (#1778)
## What issue does this PR close?
Partially address #1749.
## Rationale for this change
This PR fixes a bug in delete file loading when a `FileScanTask`
contains both positional and equality delete files. We hit this when
running Iceberg Java test suite via Comet in
apache/datafusion-comet#2528. The tests that
failed were
```
TestSparkExecutorCache > testMergeOnReadUpdate()
TestSparkExecutorCache > testMergeOnReadMerge()
TestSparkExecutorCache > testMergeOnReadDelete()
```
**The Bug:**
The condition in `try_start_eq_del_load` (delete_filter.rs:71-73) was
inverted. It returned `None` when the equality delete file was not in
the cache, causing the loader to skip loading it. When
`build_equality_delete_predicate` was later called, it would fail with
"Missing predicate for equality delete file".
## What changes are included in this PR?
**The Fix:**
- Inverted the condition so it returns `None` when the file is already
in the cache (being loaded or loaded), preventing duplicate work across
concurrent tasks
- When the file is not in the cache, mark it as Loading and proceed with
loading
**Additional Changes:**
- Added test case `test_load_deletes_with_mixed_types` that reproduces
the bug scenario
## Are these changes tested?
Yes, this PR includes a new unit test
`test_load_deletes_with_mixed_types` that:
- Creates a `FileScanTask` with both a positional delete file and an
equality delete file
- Verifies that `load_deletes` successfully processes both types
- Verifies that `build_equality_delete_predicate` succeeds without the
"Missing predicate" error
- We hit this when running Iceberg Java test suite via Comet in
apache/datafusion-comet#2528. I also confirmed
that it fixes the tests in Iceberg Java's suite.
The test would fail before this fix and passes after.
0 commit comments