DF 49 branch fix by zhuqi-lucas · Pull Request #9 · massive-com/arrow-datafusion

zhuqi-lucas · 2025-09-03T03:28:09Z

Merge upstream DF 49.0.2, and resolve confilcts
Merge df-48-stream branch and resolve confilcts.

this is too noisy and not helpful yet, we don't have a fully implemented alternative (cherry picked from commit 0183244) Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>

…ec of the correct size (apache#16995) (apache#17068) * apache#16994 Ensure CooperativeExec#maintains_input_order returns a Vec of the correct size * apache#16994 Extend default ExecutionPlan invariant checks Add checks that verify the length of the vectors returned by methods that need to return a value per child. (cherry picked from commit 2968331)

* Add ExecutionPlan::reset_state * Update datafusion/sqllogictest/test_files/cte.slt * Add reference * fmt * add to upgrade guide * add explain plan, implement in more plans * fmt * only explain --------- Co-authored-by: Robert Ream <robert@stably.io>

* Preserve equivalence properties during projection pushdown (apache#17129) * Adds parquet data diffs --------- Co-authored-by: Matthew Kim <38759997+friendlymatthew@users.noreply.github.com>

…17123) (apache#17174) * Pass the input schema to stats_projection for ProjectionExpr * Adds a test * fmt * clippy --------- Co-authored-by: Haresh Khanna <kh.haresh@gmail.com>

* fix: string_agg not respecting ORDER BY * Fix equality of parametrizable ArrayAgg function (apache#17065) The `ArrayAgg` struct is stateful, therefore it must implement `AggregateUDFImpl::equals` and `hash_value` functions. * Implement AggregateUDFImpl::equals and AggregateUDFImpl::hash_value for ArrayAgg * Implement alternative fix * Remove 'use std::any::Any' * Add sqllogictest for string_agg plan * Revert as_any to their original implementations --------- Co-authored-by: Piotr Findeisen <piotr.findeisen@gmail.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

* Update to version 49.0.1 * Add changelog for 49.0.1 * Fix sqllogictests * update configs * Update with PR * prettier * Fix slt race condition * Tweak release notes

…-features integration-tests --locked

…e#17270) Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

…17274) * fix: align `array_has` null buffer for scalar (apache#17272) * fix: align `array_has` null buffer for scalar * merge

* Update versio to 49.0.2 * Add changelog * update configuration docs

…alescePartitionsExec missed fetch)

…eserving_variants_preserves_fetch

* Simplify predicates in filter * add slt test * Use BtreeMap to make tests stable * process edge coner * add doc for simplify_predicates.rs * add as_literal to make code neat * reorgnize file * reduce clone call

…BY clause (apache#16257) * Add order by clause to limit query for consistent results * test: update explain plan

Make CI green

…_fix

zhuqi-lucas · 2025-09-03T07:36:20Z

datafusion/physical-optimizer/src/enforce_distribution.rs

+        // We can make sure that `plan` has an ordering because
+        // `SortPreservingMergeExec` requires ordering to be constructed.
+        // If there is no ordering, `SortPreservingMergeExec::new` will panic
+        let ordering = plan.output_ordering().cloned().unwrap();


Here to use unwrap(), cc @xudong963 @matthewmturner i am not sure if it's 100% safe, need another look, before DF 49, we have a Lex default to apply:

if fetch.is_some() { let ordering = plan .output_ordering() .cloned() .unwrap_or_else(LexOrdering::default); let plan = Arc::new( SortPreservingMergeExec::new(ordering, plan).with_fetch(fetch.take()), ); optimized_distribution_ctx = DistributionContext::new(plan, data, vec![optimized_distribution_ctx]); }

But after DF 49, we don't have LexOrdering::default now. So i believe, we can use unwrap directly because when we construct it, it should be there.

Can we use the spm above here? Just give it a fetch

Good point, will address it.

zhuqi-lucas · 2025-09-03T08:55:20Z

datafusion/sqllogictest/test_files/array.slt

 04)------AggregateExec: mode=Partial, gby=[], aggr=[count(Int64(1))]
 05)--------ProjectionExec: expr=[]
 06)----------CoalesceBatchesExec: target_batch_size=8192
-<<<<<<< HEAD


Here we resolve the conflicts which this branch not handling before.

xudong963 · 2025-09-04T04:55:17Z

datafusion/physical-optimizer/src/enforce_distribution.rs

-                SortPreservingMergeExec::new(ordering.clone(), Arc::clone(&input.plan))
-                    .with_fetch(fetch.take()),
+                SortPreservingMergeExec::new(req.clone(), Arc::clone(&input.plan))
+                    .with_fetch(*fetch),


Why remove the take here?

Good catch, it's a mistake change, fetch take should have better performance. Addressed in latest PR.

xudong963 · 2025-09-04T04:59:56Z

datafusion/physical-optimizer/src/enforce_distribution.rs

+        // We can make sure that `plan` has an ordering because
+        // `SortPreservingMergeExec` requires ordering to be constructed.
+        // If there is no ordering, `SortPreservingMergeExec::new` will panic
+        let ordering = plan.output_ordering().cloned().unwrap();


Can we use the spm above here? Just give it a fetch

zhuqi-lucas · 2025-09-04T05:33:07Z

Thank you @xudong963 for review, addressed comments in latest PR.

xudong963

LGTM, let's go

mbutrovich and others added 30 commits August 6, 2025 16:32

remove warning from every file open (apache#16968) (apache#17059)

9cfb9cd

this is too noisy and not helpful yet, we don't have a fully implemented alternative (cherry picked from commit 0183244) Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>

[branch-49] Backport apache#17129 to branch 49 (apache#17143)

ee28aa7

* Preserve equivalence properties during projection pushdown (apache#17129) * Adds parquet data diffs --------- Co-authored-by: Matthew Kim <38759997+friendlymatthew@users.noreply.github.com>

Pass the input schema to stats_projection for ProjectionExpr (apache#…

52e4ef8

…17123) (apache#17174) * Pass the input schema to stats_projection for ProjectionExpr * Adds a test * fmt * clippy --------- Co-authored-by: Haresh Khanna <kh.haresh@gmail.com>

[branch-49] Update version to 49.0.1 and add changelog (apache#17175)

d1a6e9a

* Update to version 49.0.1 * Add changelog for 49.0.1 * Fix sqllogictests * update configs * Update with PR * prettier * Fix slt race condition * Tweak release notes

cherry-pick inlist fix (apache#17254)

374fcec

fix check license header

930608a

fix cargo check: cargo check --profile ci --workspace --all-targets -…

66ae588

…-features integration-tests --locked

fix cargo example

292641c

FFI_RecordBatchStream was causing a memory leak (apache#17190) (apach…

a6068c2

…e#17270) Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

fix: align array_has null buffer for scalar (apache#17272) (apache#…

0d04475

…17274) * fix: align `array_has` null buffer for scalar (apache#17272) * fix: align `array_has` null buffer for scalar * merge

[branch-49] Prepare 49.0.2 version and changelog (apache#17277)

f43df3f

* Update versio to 49.0.2 * Add changelog * update configuration docs

fix cargo check --profile ci --no-default-features -p datafusion-proto

25058de

fix cargo doc

c46f7a9

fix ut:custom_sources_cases::statistics::sql_limit(with_node_id of Co…

deaf2e2

…alescePartitionsExec missed fetch)

fix ut: test_no_pushdown_through_aggregates & test_plan_with_order_pr…

f1b1bd8

…eserving_variants_preserves_fetch

fix format

7dd5e6e

fix roundtrip_test

2eca4c0

schema_force_view_types to true

8baa05d

use utfview8

9b2fbbb

schema_force_view_types to false(try true after df49)

63c2ebc

fix page_index_filter_one_col and remove an example of proto

ed718c0

fix configs.md

0bb16fa

fix clippy

09ff8f7

update configs.md

1545f2d

fix flaky test limit.test

ca5b0fb

Simplify predicates in PushDownFilter optimizer rule (apache#16362)

d8c3e03

* Simplify predicates in filter * add slt test * Use BtreeMap to make tests stable * process edge coner * add doc for simplify_predicates.rs * add as_literal to make code neat * reorgnize file * reduce clone call

Fix intermittent SQL logic test failure in limit.slt by adding ORDER …

2099882

…BY clause (apache#16257) * Add order by clause to limit query for consistent results * test: update explain plan

xudong963 and others added 8 commits August 25, 2025 15:48

fix tpch q19

2c7836a

public GroupValues & new_group_values

9191f39

fix clippy

d358db4

Merge pull request #8 from polygon-io/branch-48-stream-fix

6e71350

Make CI green

Merge remote-tracking branch 'upstream/branch-49' into branch_49_fix

c6b8211

Merge remote-tracking branch 'origin/branch-48-stream' into branch_49…

5a99099

…_fix

fix fetch with new order lex

cefa63a

fix fetch add back with new lex order

1f47d46

github-actions bot added documentation Improvements or additions to documentation physical-expr development-process optimizer core sqllogictest ffi physical-plan datasource functions proto labels Sep 3, 2025

zhuqi-lucas added 4 commits September 3, 2025 11:42

fix clippy

70a3c94

add order needed

a93e81e

fix

6a3d4f8

fix auth check and port upstream fix: apache#17355

91e2904

zhuqi-lucas commented Sep 3, 2025

View reviewed changes

xudong963 reviewed Sep 4, 2025

View reviewed changes

Addressed in latest PR

1a2f8dc

xudong963 approved these changes Sep 4, 2025

View reviewed changes

xudong963 merged commit be0276d into branch-49 Sep 4, 2025
55 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DF 49 branch fix#9

DF 49 branch fix#9
xudong963 merged 44 commits intobranch-49from
branch_49_fix

zhuqi-lucas commented Sep 3, 2025

Uh oh!

zhuqi-lucas Sep 3, 2025 •

edited

Loading

Uh oh!

xudong963 Sep 4, 2025

Uh oh!

zhuqi-lucas Sep 4, 2025

Uh oh!

zhuqi-lucas Sep 3, 2025

Uh oh!

xudong963 Sep 4, 2025

Uh oh!

zhuqi-lucas Sep 4, 2025

Uh oh!

xudong963 Sep 4, 2025

Uh oh!

zhuqi-lucas commented Sep 4, 2025

Uh oh!

xudong963 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

Conversation

zhuqi-lucas commented Sep 3, 2025

Uh oh!

zhuqi-lucas Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xudong963 Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

zhuqi-lucas Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

zhuqi-lucas Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

xudong963 Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

zhuqi-lucas Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

xudong963 Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

zhuqi-lucas commented Sep 4, 2025

Uh oh!

xudong963 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

zhuqi-lucas Sep 3, 2025 •

edited

Loading