allow filters referencing parquet struct columns to be pushed down #19556
+484
−806
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
sqllogictest
github-actions
bot
added
common
adriangb
commented
Jan 7, 2026
| ScalarValue::try_new_null(&DataType::Map(map_field_ref, false)).unwrap(), | ||
| ScalarValue::try_new_null(&DataType::Union( | ||
| UnionFields::new(vec![42], vec![field_ref]), | ||
| UnionFields::try_new(vec![42], vec![field_ref]).unwrap(), |
Contributor
Author
There was a problem hiding this comment.
Note that this is just handling upstream changes in arrow, not actually related to this PR
adriangb
force-pushed
the
parquet-struct-filter
branch
2 times, most recently
from
8b0bd72 to
7568ded
Compare
this does *not* benchmark (or implement) reading a single field of a struct column. this only tests being able to late materialize a struct columns *as a whole*
adriangb
force-pushed
the
parquet-struct-filter
branch
from
7568ded to
39b37af
Compare
Contributor
Author
|
There's some test failures but I think those are all mostly unrelated to this change and are issues that need fixing with the update to arrow-rs/main. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A step towards #11745.
Depends on upstream arrow fixes..
Note that this does not do anything fancy with parquet struct access like actually generating the leaf mask to read only some fields, it reads the entire struct column and process it in memory much like currently happens. That should be an easy followup though.
Benchmark numbers: