Skip to content

Commit 082de7f

Browse files
baibaichenyaooqinn
authored andcommitted
[SPARK-54850][SQL] Improve extractShuffleIds to find AdaptiveSparkPlanExec anywhere in plan tree
### What changes were proposed in this pull request? This PR uses `collectFirst` to find the first `AdaptiveSparkPlanExec` node anywhere in the plan tree, instead of assuming the root plan is an `AdaptiveSparkPlanExec`. ### Why are the changes needed? #52157 introduced the `extractShuffleIds` method in `SQLExecution` to find shuffle IDs of `SparkPlan`. Previously, the method implicitly assumed that if AQE is enabled, the `AdaptiveSparkPlanExec` would be at the root of the input. Since Spark only inserts `AdaptiveSparkPlanExec` under Command, this assumption was fine. However, the `AdaptiveSparkPlanExec` may not be the root node in Gluten. Gluten needs to insert a special physical plan to do column to row transition. By using `collectFirst`, we can correctly locate the `AdaptiveSparkPlanExec` regardless of its position in the plan tree, which improves compatibility. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Pass GHA. ### Was this patch authored or co-authored using generative AI tooling? No Closes #53620 from baibaichen/feature/extractShuffleIds. Authored-by: Chang chen <[email protected]> Signed-off-by: Kent Yao <[email protected]>
1 parent 2afc713 commit 082de7f

File tree

1 file changed

+4
-3
lines changed

1 file changed

+4
-3
lines changed

sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -71,11 +71,12 @@ object SQLExecution extends Logging {
7171
}
7272

7373
private def extractShuffleIds(plan: SparkPlan): Seq[Int] = {
74-
plan match {
74+
val shuffleIdsOption = plan.collectFirst {
7575
case ae: AdaptiveSparkPlanExec =>
7676
ae.context.shuffleIds.asScala.keys.toSeq
77-
case nonAdaptivePlan =>
78-
nonAdaptivePlan.collect {
77+
}
78+
shuffleIdsOption.getOrElse {
79+
plan.collect {
7980
case exec: ShuffleExchangeLike => exec.shuffleId
8081
}
8182
}

0 commit comments

Comments
 (0)