Skip to content

fix: Distribution error failing in the SanityCheck, for a specific influxql plan. #58

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wiedld wants to merge 4 commits into from
Closed

fix: Distribution error failing in the SanityCheck, for a specific influxql plan. #58

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
90280ea
test: recreating the iox plan:
wiedld Feb 7, 2025
f792cfa
test: reproducer of SanityCheck failure after EnforceSorting removes …
wiedld Feb 12, 2025
202860b
fix: special case to not remove the needed coalesce
wiedld Feb 12, 2025
1c10b8b
fix(ci): build error with wasm (#14494)
Lordworms Feb 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 11 additions & 12 deletions datafusion/core/src/physical_optimizer/enforce_sorting.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@

use std::sync::Arc;

use super::utils::{add_sort_above, add_sort_above_with_check};
use super::utils::{add_sort_above, add_sort_above_with_check, is_aggregation};
use crate::config::ConfigOptions;
use crate::error::Result;
use crate::physical_optimizer::replace_with_order_preserving_variants::{
Expand Down Expand Up @@ -516,7 +516,7 @@ fn remove_bottleneck_in_subplan(
) -> Result<PlanWithCorrespondingCoalescePartitions> {
let plan = &requirements.plan;
let children = &mut requirements.children;
if is_coalesce_partitions(&children[0].plan) {
if is_coalesce_partitions(&children[0].plan) && !is_aggregation(plan) {
Copy link
Author

@wiedld wiedld Feb 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the fix, for now.

A proper fix (if we decide to fix at this conditional) should be comparing the partitioning needs of the parent (of coalesce) vs children of coalesce. My initial attempt to do so caused other DF tests to fail.

I didn't proceed further since I'm unsure of the correct solution at a higher level.

Copy link
Collaborator

@alamb alamb Feb 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be checking that it is an aggregation and that that the aggregation mode is SinglePartitioned 🤔

// We can safely use the 0th index since we have a `CoalescePartitionsExec`.
let mut new_child_node = children[0].children.swap_remove(0);
while new_child_node.plan.output_partitioning() == plan.output_partitioning()
Expand Down Expand Up @@ -2261,21 +2261,20 @@ mod tests {
get_plan_string(&optimized),
vec![
"SortPreservingMergeExec: [a@0 ASC]",
" SortExec: expr=[a@0 ASC], preserve_partitioning=[true]",
" SortExec: expr=[a@0 ASC], preserve_partitioning=[false]",
" AggregateExec: mode=SinglePartitioned, gby=[a@0 as a1], aggr=[]",
" ProjectionExec: expr=[a@0 as a, b@1 as value]",
" UnionExec",
" ParquetExec: file_groups={1 group: [[x]]}, projection=[a, b, c, d, e]",
" ParquetExec: file_groups={1 group: [[x]]}, projection=[a, b, c, d, e]",
" CoalescePartitionsExec",
" ProjectionExec: expr=[a@0 as a, b@1 as value]",
" UnionExec",
" ParquetExec: file_groups={1 group: [[x]]}, projection=[a, b, c, d, e]",
" ParquetExec: file_groups={1 group: [[x]]}, projection=[a, b, c, d, e]",
],
);

// Plan is now invalid.
// Plan is valid.
let checker = SanityCheckPlan::new();
let err = checker
.optimize(optimized, &Default::default())
.unwrap_err();
assert!(err.message().contains(" does not satisfy distribution requirements: HashPartitioned[[a@0]]). Child-0 output partitioning: UnknownPartitioning(2)"));
let checker = checker.optimize(optimized, &Default::default());
Copy link
Collaborator

@alamb alamb Feb 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like the idea of running the SanityChecker as part of the tests

assert!(checker.is_ok());

Ok(())
}
Expand Down
5 changes: 5 additions & 0 deletions datafusion/core/src/physical_optimizer/utils.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,3 +109,8 @@ pub fn is_union(plan: &Arc<dyn ExecutionPlan>) -> bool {
pub fn is_repartition(plan: &Arc<dyn ExecutionPlan>) -> bool {
plan.as_any().is::<RepartitionExec>()
}

/// Checks whether the given operator is a [`AggregateExec`].
pub fn is_aggregation(plan: &Arc<dyn ExecutionPlan>) -> bool {
plan.as_any().is::<AggregateExec>()
}
Loading