Skip to content

WIP: Upgrade DataFusion to arrow-rs/parquet 58.0.0 #19728

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
alamb wants to merge 16 commits into
base: main
Choose a base branch
from
Draft

WIP: Upgrade DataFusion to arrow-rs/parquet 58.0.0 #19728

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
120d2b0
Update to pre-release arrow
alamb Dec 16, 2025
40a6a71
Fix clippy due to UnionFields::new is deprecated
alamb Dec 16, 2025
f50b9c3
Fix real bug in test
alamb Dec 16, 2025
87563d8
Update test for fixed issue upstream
alamb Dec 16, 2025
4afd46f
Update tests for new feature
alamb Dec 18, 2025
70e2841
Update tests for struct casting
alamb Dec 18, 2025
b29b662
Update to latest
alamb Jan 7, 2026
6a2e631
Merge remote-tracking branch 'apache/main' into alamb/update_arrow_57…
alamb Jan 7, 2026
ca11966
Merge branch 'main' into alamb/update_arrow_57.2.0
alamb Jan 8, 2026
63a85eb
Update to pre-release main
alamb Jan 10, 2026
277aac4
Update expected value of make_interval
alamb Jan 10, 2026
bb29b0c
Update expected output
alamb Jan 10, 2026
35b97fa
Update to latest arrow
alamb Jan 10, 2026
d2e8da6
Update rev
Dandandan Jan 16, 2026
1002408
update outputs
Dandandan Jan 16, 2026
c062756
Merge
Dandandan Jan 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 16 additions & 33 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

29 changes: 29 additions & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -271,3 +271,32 @@ incremental = false
inherits = "release"
debug = true
strip = false

## Temporary arrow-rs patch until 58 is released

[patch.crates-io]
arrow = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
arrow-array = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
arrow-buffer = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
arrow-cast = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
arrow-data = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
arrow-ipc = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
arrow-schema = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
arrow-select = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
arrow-string = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
arrow-ord = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
arrow-flight = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
parquet = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }

#arrow = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow" }
#arrow-array = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow-array" }
#arrow-buffer = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow-buffer" }
#arrow-cast = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow-cast" }
#arrow-data = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow-data" }
#arrow-ipc = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow-ipc" }
#arrow-schema = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow-schema" }
#arrow-select = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow-select" }
#arrow-string = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow-string" }
#arrow-ord = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow-ord" }
#arrow-flight = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow-flight" }
#parquet = { path= "/Users/andrewlamb/Software/arrow-rs2/parquet" }
8 changes: 4 additions & 4 deletions datafusion-cli/src/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -617,8 +617,8 @@ mod tests {
| filename | file_size_bytes | metadata_size_bytes | hits | extra |
+-----------------------------------+-----------------+---------------------+------+------------------+
| alltypes_plain.parquet | 1851 | 8882 | 2 | page_index=false |
| alltypes_tiny_pages.parquet | 454233 | 269266 | 2 | page_index=true |
| lz4_raw_compressed_larger.parquet | 380836 | 1347 | 2 | page_index=false |
| alltypes_tiny_pages.parquet | 454233 | 269074 | 2 | page_index=true |
Copy link
Contributor Author

@alamb alamb Jan 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this reduction in metadata size is a direct consequence of @WaterWhisperer's PR to improve PageEncoding representation

| lz4_raw_compressed_larger.parquet | 380836 | 1339 | 2 | page_index=false |
+-----------------------------------+-----------------+---------------------+------+------------------+
");

Expand Down Expand Up @@ -648,8 +648,8 @@ mod tests {
| filename | file_size_bytes | metadata_size_bytes | hits | extra |
+-----------------------------------+-----------------+---------------------+------+------------------+
| alltypes_plain.parquet | 1851 | 8882 | 5 | page_index=false |
| alltypes_tiny_pages.parquet | 454233 | 269266 | 2 | page_index=true |
| lz4_raw_compressed_larger.parquet | 380836 | 1347 | 3 | page_index=false |
| alltypes_tiny_pages.parquet | 454233 | 269074 | 2 | page_index=true |
| lz4_raw_compressed_larger.parquet | 380836 | 1339 | 3 | page_index=false |
+-----------------------------------+-----------------+---------------------+------+------------------+
");

Expand Down
4 changes: 1 addition & 3 deletions datafusion/sqllogictest/test_files/array.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4747,10 +4747,8 @@ select array_union(arrow_cast([], 'LargeList(Int64)'), arrow_cast([], 'LargeList
[]

# array_union scalar function #7
query ?
query error DataFusion error: Arrow error: Invalid argument error: Codecs \[List\(RowConverter \{ fields: \[SortField \{ options: SortOptions \{ descending: false, nulls_first: true \}, data_type: Null \}\], codecs: \[Stateless\] \}\)\] did not consume all bytes for row 0, remaining bytes: \[1\]
select array_union([[null]], []);
----
[[]]

query error DataFusion error: Error during planning: Failed to coerce arguments to satisfy a call to 'array_union' function:
select array_union(arrow_cast([[null]], 'LargeList(List(Int64))'), arrow_cast([], 'LargeList(Int64)'));
Expand Down
1 change: 0 additions & 1 deletion datafusion/sqllogictest/test_files/datetime/arith_date_time.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,4 +113,3 @@ SELECT '2001-09-28'::date / '03:00'::time

query error Invalid timestamp arithmetic operation
SELECT '2001-09-28'::date % '03:00'::time

2 changes: 1 addition & 1 deletion datafusion/sqllogictest/test_files/datetime/arith_timestamp_duration.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,4 +144,4 @@ query error Invalid timestamp arithmetic operation
SELECT '2001-09-28T01:00:00'::timestamp % arrow_cast(12345, 'Duration(Second)');

query error Invalid timestamp arithmetic operation
SELECT '2001-09-28T01:00:00'::timestamp / arrow_cast(12345, 'Duration(Second)');
SELECT '2001-09-28T01:00:00'::timestamp / arrow_cast(12345, 'Duration(Second)');
14 changes: 7 additions & 7 deletions datafusion/sqllogictest/test_files/explain_tree.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -268,7 +268,7 @@ physical_plan
06)┌─────────────┴─────────────┐
07)│ DataSourceExec │
08)│ -------------------- │
09)│ bytes: 1040 │
09)│ bytes: 1024 │
10)│ format: memory │
11)│ rows: 2 │
12)└───────────────────────────┘
Expand Down Expand Up @@ -345,7 +345,7 @@ physical_plan
15)┌─────────────┴─────────────┐┌─────────────┴─────────────┐
16)│ DataSourceExec ││ ProjectionExec │
17)│ -------------------- ││ -------------------- │
18)│ bytes: 520 ││ date_col: date_col │
18)│ bytes: 512 ││ date_col: date_col │
19)│ format: memory ││ int_col: int_col │
20)│ rows: 1 ││ │
21)│ ││ string_col: │
Expand Down Expand Up @@ -592,7 +592,7 @@ physical_plan
07)┌─────────────┴─────────────┐
08)│ DataSourceExec │
09)│ -------------------- │
10)│ bytes: 520 │
10)│ bytes: 512 │
11)│ format: memory │
12)│ rows: 1 │
13)└───────────────────────────┘
Expand Down Expand Up @@ -954,7 +954,7 @@ physical_plan
13)┌─────────────┴─────────────┐
14)│ DataSourceExec │
15)│ -------------------- │
16)│ bytes: 520 │
16)│ bytes: 512 │
17)│ format: memory │
18)│ rows: 1 │
19)└───────────────────────────┘
Expand Down Expand Up @@ -1305,7 +1305,7 @@ physical_plan
42)┌─────────────┴─────────────┐┌─────────────┴─────────────┐
43)│ DataSourceExec ││ DataSourceExec │
44)│ -------------------- ││ -------------------- │
45)│ bytes: 296 ││ bytes: 288 │
45)│ bytes: 288 ││ bytes: 280 │
46)│ format: memory ││ format: memory │
47)│ rows: 1 ││ rows: 1 │
48)└───────────────────────────┘└───────────────────────────┘
Expand All @@ -1324,14 +1324,14 @@ physical_plan
04)┌─────────────┴─────────────┐┌─────────────┴─────────────┐
05)│ DataSourceExec ││ ProjectionExec │
06)│ -------------------- ││ -------------------- │
07)│ bytes: 296 ││ id: CAST(id AS Int32) │
07)│ bytes: 288 ││ id: CAST(id AS Int32) │
08)│ format: memory ││ name: name │
09)│ rows: 1 ││ │
10)└───────────────────────────┘└─────────────┬─────────────┘
11)-----------------------------┌─────────────┴─────────────┐
12)-----------------------------│ DataSourceExec │
13)-----------------------------│ -------------------- │
14)-----------------------------│ bytes: 288 │
14)-----------------------------│ bytes: 280 │
15)-----------------------------│ format: memory │
16)-----------------------------│ rows: 1 │
17)-----------------------------└───────────────────────────┘
Expand Down
4 changes: 2 additions & 2 deletions datafusion/sqllogictest/test_files/spark/datetime/make_interval.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,15 +96,15 @@ NULL
query T
SELECT make_interval(0, 0, 0, 0, 0, 0, 0.0) || '';
----
Copy link
Contributor Author

@alamb alamb Jan 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is due to @Jefffrey 's improvement in

Copy link
Contributor

@davidlghellin davidlghellin Jan 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

close #17455

Copy link
Contributor

@Jefffrey Jefffrey Jan 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably should remove the accompanying comments too, these

# Intervals being rendered as empty string, see issue:
# https://github.com/apache/datafusion/issues/17455
# We expect something like 0.00 secs with query ?

(empty)
0 secs

# Intervals being rendered as empty string, see issue:
# https://github.com/apache/datafusion/issues/17455
# We expect something like 0.00 secs with query ?
query T
SELECT make_interval() || '';
----
(empty)
0 secs

query ?
SELECT INTERVAL '1' SECOND AS iv;
Expand Down
6 changes: 4 additions & 2 deletions datafusion/sqllogictest/test_files/window.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6068,7 +6068,7 @@ FROM (
# `WHERE acctbal > ( SELECT AVG(acctbal) FROM suppliers)` into a Join,
# breaking the input schema passed to the window function above.
# See: https://github.com/apache/datafusion/issues/17770
query I
query error
WITH suppliers AS (
SELECT *
FROM (VALUES (1, 10.0), (1, 20.0)) AS t(nation, acctbal)
Expand All @@ -6080,4 +6080,6 @@ WHERE acctbal > (
SELECT AVG(acctbal) FROM suppliers
);
----
1
DataFusion error: Join Error
Copy link
Contributor

@Dandandan Dandandan Jan 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This somehow errors now 🤔

caused by
External error: task 20768 panicked with message "assertion `left == right` failed\n left: 4\n right: 1"
Loading