Scan Delete Support Part 4: Delete File Loading; Skeleton for Processing

[sdd]

Extends the DeleteFileManager introduced in #950 To include loading of delete files, storage and retrieval of parsed delete files from shared state, and the outline for how parsing will connect up to this new work.

Issue: #630

[jonathanc-n]

This is nice, will look at the parsed records next.

[sdd]

@liurenjie1024, @Xuanwo, @Fokko - this is ready for re-review, if you could take a look that would be great!

[liurenjie1024]

Thanks @sdd for this pr. There are some missing points in current design. Also I would suggest not putting too much in DeleteFilterManager. I suppose DeleterFilterManager acting more like a delete loader, which manages the io and caching of record batch. The actual filtering part, could delegate to DeleteFilter, WDYT? I think a good reference implementation is java's DeleteFilter, see https://github.com/apache/iceberg/blob/af8e3f5a40f4f36bbe1d868146749e2341471586/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L50

[sdd]

Thanks for the review @liurenjie1024 - much appreciated. Will come back with a revised design.

[liurenjie1024]

Let's wait for a moment to merge it after 0.5.0 release

[sdd]

Hi @liurenjie1024 / @Xuanwo / @xxchan.

This is now ready again for review after a refactor taking into account @xxchan's great feedback. I'll be on holiday for a week after today so it would be great if you guys could take a look. Thanks!

[liurenjie1024]

Thanks @sdd for this pr, I think it's fine to move forward for now.