Skip to content

Support Shredded Lists/Array in variant_get #8354

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sdf-jkl wants to merge 49 commits into
base: main
Choose a base branch
from
+427 −20
Open

Support Shredded Lists/Array in variant_get #8354

Changes from 42 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
9c25cc4
Add test shredded variant list array
sdf-jkl Sep 15, 2025
ed961a4
Add basic tests
sdf-jkl Sep 16, 2025
03ecb95
Merge branch 'apache:main' into shredded_list_support
sdf-jkl Sep 16, 2025
158d6d7
Merge branch 'apache:main' into shredded_list_support
sdf-jkl Sep 16, 2025
d53c831
Redo test shredded array
sdf-jkl Sep 17, 2025
174e429
Merge branch 'main' of https://github.com/apache/arrow-rs into shredd…
sdf-jkl Sep 18, 2025
69de7d7
Rebuild the shredded list array
sdf-jkl Sep 19, 2025
cc6d787
Use select::take to build the output array
sdf-jkl Sep 23, 2025
8f6ad1b
Merge branch 'main' of https://github.com/apache/arrow-rs into shredd…
sdf-jkl Sep 23, 2025
bc8abd9
Merge branch 'main' of https://github.com/apache/arrow-rs into shredd…
sdf-jkl Sep 25, 2025
c0d2065
Pass one test
sdf-jkl Sep 25, 2025
85aaa3f
Merge branch 'main' of https://github.com/apache/arrow-rs into shredd…
sdf-jkl Sep 25, 2025
40b6311
Get typed values directly
sdf-jkl Sep 25, 2025
f6e88ef
Added support for utf8, largeUtf8, utf8view
sdf-jkl Oct 13, 2025
61ed178
added tests for utf8, largeUtf8, utf8view
sdf-jkl Oct 13, 2025
1fb612d
fix tests
sdf-jkl Oct 13, 2025
2b6d280
Update parquet-variant-compute/src/variant_to_arrow.rs
sdf-jkl Oct 14, 2025
398b52d
Merge branch 'main' into variant_to_arrow_utf8
sdf-jkl Oct 14, 2025
defa07b
Update parquet-variant-compute/src/variant_to_arrow.rs
sdf-jkl Oct 20, 2025
5022acd
Support LargeUtf8, Utf8-View
sdf-jkl Oct 20, 2025
ed66007
Merge branch 'main' into variant_to_arrow_utf8
sdf-jkl Oct 20, 2025
196b5d4
Fix Merge errors
sdf-jkl Oct 20, 2025
642d192
Update arrow-array/src/builder/generic_bytes_builder.rs
sdf-jkl Oct 20, 2025
76b3c80
Add docs for AVERAGE_STRING_LENGTH const
sdf-jkl Oct 21, 2025
35785d6
Merge branch 'variant_to_arrow_utf8' of https://github.com/sdf-jkl/ar…
sdf-jkl Oct 21, 2025
5914218
cargo fmt
sdf-jkl Oct 21, 2025
216d401
cargo fmt
sdf-jkl Oct 21, 2025
3aa6cf3
Merge branch 'variant_to_arrow_utf8' into shredded_list_support
sdf-jkl Oct 22, 2025
15fc8be
Merge branch 'main' of https://github.com/apache/arrow-rs into shredd…
sdf-jkl Oct 22, 2025
04b9941
Quick fix variant_get
sdf-jkl Oct 24, 2025
857f0e2
Merge branch 'main' into shredded_list_support
sdf-jkl Oct 24, 2025
1edfeca
Merge branch 'main' into shredded_list_support
sdf-jkl Nov 11, 2025
6d6793d
fix merge errors
sdf-jkl Nov 11, 2025
da528c2
Merge branch 'main' of https://github.com/apache/arrow-rs into shredd…
sdf-jkl Dec 3, 2025
71412b8
Merge branch 'main' into shredded_list_support
sdf-jkl Dec 3, 2025
0c32647
Merge branch 'main' of https://github.com/apache/arrow-rs into shredd…
sdf-jkl Feb 19, 2026
5b899d8
Merge branch 'shredded_list_support' of https://github.com/sdf-jkl/ar…
sdf-jkl Feb 19, 2026
9cd01d2
Simplify tests using shred_variant
sdf-jkl Feb 20, 2026
cecd39f
Add tests suggested by @klion26
sdf-jkl Feb 21, 2026
a776982
Fix typed and untyped values logic
sdf-jkl Feb 21, 2026
cfe7c00
Add support for LargeListArray + OBB err when safe_cast
sdf-jkl Feb 21, 2026
fc99bf0
Use ShreddingState instead of BorrowedShreddingState in ShreddedPathS…
sdf-jkl Feb 21, 2026
ccbf59b
Reuse ShreddingState methods
sdf-jkl Feb 23, 2026
cbfa058
nit fix
sdf-jkl Feb 25, 2026
cf94d43
use else if chain
sdf-jkl Feb 25, 2026
91589ad
add cast_options.safe docs
sdf-jkl Feb 25, 2026
e8e7fb1
Merge branch 'main' of https://github.com/apache/arrow-rs into shredd…
sdf-jkl Feb 26, 2026
28ec53c
support list-like arrays
sdf-jkl Feb 26, 2026
279b634
match typed value instead of donwcast attempts
sdf-jkl Mar 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Loading