Skip to content

Commit 8b1019a

Browse files
committed
chore: add docs to update_coalesce_ctx_children
1 parent 1e7bc76 commit 8b1019a

File tree

1 file changed

+23
-3
lines changed
  • datafusion/physical-optimizer/src/enforce_sorting

1 file changed

+23
-3
lines changed

datafusion/physical-optimizer/src/enforce_sorting/mod.rs

Lines changed: 23 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -153,10 +153,8 @@ fn is_coalesce_to_remove(
153153
node.as_any()
154154
.downcast_ref::<CoalescePartitionsExec>()
155155
.map(|_coalesce| {
156-
// TODO(wiedld): find a more generalized approach that does not rely on
157-
// pattern matching the structure of the DAG
158156
// Note that the `Partitioning::satisfy()` (parent vs. coalesce.child) cannot be used for cases of:
159-
// * Repartition -> Coalesce -> Repartition
157+
// * Repartition -> Coalesce -> Repartition
160158

161159
let parent_req_single_partition = matches!(
162160
parent.required_input_distribution()[0],
@@ -173,6 +171,28 @@ fn is_coalesce_to_remove(
173171
.unwrap_or(false)
174172
}
175173

174+
/// Discovers the linked Coalesce->Sort cascades.
175+
///
176+
/// This linkage is used in [`remove_bottleneck_in_subplan`] to selectively
177+
/// remove the linked coalesces in the subplan. Then afterwards, an SPM is added
178+
/// at the root of the subplan in order to parallelize sorts.
179+
/// Refer to the [`parallelize_sorts`] for more details on sort parallelization.
180+
///
181+
/// Example of linked Coalesce->Sort:
182+
/// ```text
183+
/// SortExec ctx.data=false, to halt remove_bottleneck_in_subplan)
184+
/// ...nodes... ctx.data=true (e.g. are linked in cascade)
185+
/// Coalesce ctx.data=true (e.g. is a coalesce)
186+
/// ```
187+
///
188+
/// The link should be broken (and the coalesce not removed) if the distribution
189+
/// is changed between the Coalesce->Sort cascade. Example:
190+
/// ```text
191+
/// SortExec ctx.data=false, to halt remove_bottleneck_in_subplan)
192+
/// AggregateExec ctx.data=false, to stop the link
193+
/// ...nodes... ctx.data=true (e.g. are linked in cascade)
194+
/// Coalesce ctx.data=true (e.g. is a coalesce)
195+
/// ```
176196
fn update_coalesce_ctx_children(
177197
coalesce_context: &mut PlanWithCorrespondingCoalescePartitions,
178198
) {

0 commit comments

Comments
 (0)