Commit fd08916
authored
fix(reader): Support both position and equality delete files on the same FileScanTask (#1778)
## What issue does this PR close?
Partially address #1749.
## Rationale for this change
This PR fixes a bug in delete file loading when a `FileScanTask`
contains both positional and equality delete files. We hit this when
running Iceberg Java test suite via Comet in
apache/datafusion-comet#2528. The tests that
failed were
```
TestSparkExecutorCache > testMergeOnReadUpdate()
TestSparkExecutorCache > testMergeOnReadMerge()
TestSparkExecutorCache > testMergeOnReadDelete()
```
**The Bug:**
The condition in `try_start_eq_del_load` (delete_filter.rs:71-73) was
inverted. It returned `None` when the equality delete file was not in
the cache, causing the loader to skip loading it. When
`build_equality_delete_predicate` was later called, it would fail with
"Missing predicate for equality delete file".
## What changes are included in this PR?
**The Fix:**
- Inverted the condition so it returns `None` when the file is already
in the cache (being loaded or loaded), preventing duplicate work across
concurrent tasks
- When the file is not in the cache, mark it as Loading and proceed with
loading
**Additional Changes:**
- Added test case `test_load_deletes_with_mixed_types` that reproduces
the bug scenario
## Are these changes tested?
Yes, this PR includes a new unit test
`test_load_deletes_with_mixed_types` that:
- Creates a `FileScanTask` with both a positional delete file and an
equality delete file
- Verifies that `load_deletes` successfully processes both types
- Verifies that `build_equality_delete_predicate` succeeds without the
"Missing predicate" error
- We hit this when running Iceberg Java test suite via Comet in
apache/datafusion-comet#2528. I also confirmed
that it fixes the tests in Iceberg Java's suite.
The test would fail before this fix and passes after.1 parent b6dc4cf commit fd08916
File tree
2 files changed
+116
-1
lines changed- crates/iceberg/src/arrow
2 files changed
+116
-1
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
788 | 788 | | |
789 | 789 | | |
790 | 790 | | |
| 791 | + | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
| 859 | + | |
| 860 | + | |
| 861 | + | |
| 862 | + | |
| 863 | + | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
791 | 904 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
68 | 68 | | |
69 | 69 | | |
70 | 70 | | |
71 | | - | |
| 71 | + | |
| 72 | + | |
72 | 73 | | |
73 | 74 | | |
74 | 75 | | |
| 76 | + | |
75 | 77 | | |
76 | 78 | | |
77 | 79 | | |
| |||
0 commit comments