Skip to content

Commit a5a3b7c

Browse files
Emma-82wangyum
authored andcommitted
[SPARK-52724][SQL] Enhance broadcast join OOM error handling with SHUFFLE_MERGE hint support
### What changes were proposed in this pull request? This PR enhances broadcast join OOM error handling by use shuffle sort merge join(`SHUFFLE_MERGE`) hint as a workaround for broadcast join OOM issues. ### Why are the changes needed? Reduce support workload by involving customers to diagnose and resolve broadcast join OOM issues independently. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual test: ``` Not enough memory to build and broadcast the table to all worker nodes. As a workaround, you can either disable broadcast by setting spark.sql.autoBroadcastJoinThreshold to -1 or increase the spark driver memory by setting spark.driver.memory to a higher value or analyze these tables through: `ANALYZE TABLE `t1` COMPUTE STATISTICS; ANALYZE TABLE `t2` COMPUTE STATISTICS;` or apply the shuffle sort merge join hint as described in the Spark documentation: https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-hints.html#join-hints. Not enough memory to build and broadcast the table to all worker nodes. As a workaround, you can either disable broadcast by setting spark.sql.autoBroadcastJoinThreshold to -1 or increase the spark driver memory by setting spark.driver.memory to a higher value or apply the shuffle sort merge join hint as described in the Spark documentation: https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-hints.html#join-hints. ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #51417 from Emma-82/SPARK-52724. Authored-by: emmazhang <[email protected]> Signed-off-by: Yuming Wang <[email protected]>
1 parent 9677b44 commit a5a3b7c

File tree

2 files changed

+3
-3
lines changed

2 files changed

+3
-3
lines changed

common/utils/src/main/resources/error/error-conditions.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8688,7 +8688,7 @@
86888688
},
86898689
"_LEGACY_ERROR_TEMP_2250" : {
86908690
"message" : [
8691-
"Not enough memory to build and broadcast the table to all worker nodes. As a workaround, you can either disable broadcast by setting <autoBroadcastJoinThreshold> to -1 or increase the spark driver memory by setting <driverMemory> to a higher value<analyzeTblMsg>"
8691+
"Not enough memory to build and broadcast the table to all worker nodes. As a workaround, you can either disable broadcast by setting <autoBroadcastJoinThreshold> to -1 or increase the spark driver memory by setting <driverMemory> to a higher value<analyzeTblMsg> or apply the shuffle sort merge join hint as described in the Spark documentation: https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-hints.html#join-hints."
86928692
]
86938693
},
86948694
"_LEGACY_ERROR_TEMP_2251" : {

sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2106,9 +2106,9 @@ private[sql] object QueryExecutionErrors extends QueryErrorsBase with ExecutionE
21062106
oe: OutOfMemoryError, tables: Seq[TableIdentifier]): Throwable = {
21072107
val analyzeTblMsg = if (tables.nonEmpty) {
21082108
" or analyze these tables through: " +
2109-
s"${tables.map(t => s"ANALYZE TABLE $t COMPUTE STATISTICS;").mkString(" ")}."
2109+
s"`${tables.map(t => s"ANALYZE TABLE $t COMPUTE STATISTICS;").mkString(" ")}`"
21102110
} else {
2111-
"."
2111+
""
21122112
}
21132113
new SparkException(
21142114
errorClass = "_LEGACY_ERROR_TEMP_2250",

0 commit comments

Comments
 (0)