Commit a5a3b7c
[SPARK-52724][SQL] Enhance broadcast join OOM error handling with SHUFFLE_MERGE hint support
### What changes were proposed in this pull request?
This PR enhances broadcast join OOM error handling by use shuffle sort merge join(`SHUFFLE_MERGE`) hint as a workaround for broadcast join OOM issues.
### Why are the changes needed?
Reduce support workload by involving customers to diagnose and resolve broadcast join OOM issues independently.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual test:
```
Not enough memory to build and broadcast the table to all worker nodes. As a workaround, you can either disable broadcast by setting spark.sql.autoBroadcastJoinThreshold to -1 or increase the spark driver memory by setting spark.driver.memory to a higher value or analyze these tables through: `ANALYZE TABLE `t1` COMPUTE STATISTICS; ANALYZE TABLE `t2` COMPUTE STATISTICS;` or apply the shuffle sort merge join hint as described in the Spark documentation: https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-hints.html#join-hints.
Not enough memory to build and broadcast the table to all worker nodes. As a workaround, you can either disable broadcast by setting spark.sql.autoBroadcastJoinThreshold to -1 or increase the spark driver memory by setting spark.driver.memory to a higher value or apply the shuffle sort merge join hint as described in the Spark documentation: https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-hints.html#join-hints.
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #51417 from Emma-82/SPARK-52724.
Authored-by: emmazhang <[email protected]>
Signed-off-by: Yuming Wang <[email protected]>1 parent 9677b44 commit a5a3b7c
File tree
2 files changed
+3
-3
lines changed- common/utils/src/main/resources/error
- sql/catalyst/src/main/scala/org/apache/spark/sql/errors
2 files changed
+3
-3
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8688 | 8688 | | |
8689 | 8689 | | |
8690 | 8690 | | |
8691 | | - | |
| 8691 | + | |
8692 | 8692 | | |
8693 | 8693 | | |
8694 | 8694 | | |
| |||
Lines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2106 | 2106 | | |
2107 | 2107 | | |
2108 | 2108 | | |
2109 | | - | |
| 2109 | + | |
2110 | 2110 | | |
2111 | | - | |
| 2111 | + | |
2112 | 2112 | | |
2113 | 2113 | | |
2114 | 2114 | | |
| |||
0 commit comments