Skip to content

Commit 59a0eed

Browse files
authored
Update apache-spark-autoscale.md
Proposed a clarification based on the example of auto-scaling discussed by the Synapse Discussion team( email group). The example has the purpose to bring clarification to the configuration that sometimes causes doubts in customers. ( email follow to the page owner) "In this example, if your job only requires 2 executors, it will only use 2 executors. When the job requires more, it will scale up to 6 executors (1 driver, 5 executors). When the job does not need the executors, then it will decommission the executors and if it does not need the node, it will free up the node.
1 parent edeeacc commit 59a0eed

File tree

1 file changed

+6
-1
lines changed

1 file changed

+6
-1
lines changed

articles/synapse-analytics/spark/apache-spark-autoscale.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -74,7 +74,12 @@ Apache Spark enables configuration of Dynamic Allocation of Executors through co
7474
```
7575
The defaults specified through the code override the values set through the user interface.
7676

77-
On enabling Dynamic allocation, Executors scale up or down based on the utilization of the Executors. This ensure that the Executors are provisioned in accordance with the needs of the job being run.
77+
In this example, if your job only requires 2 executors, it will only use 2 executors. When the job requires more, it will scale up to 6 executors (1 driver, 6 executors). When the job does not need the executors, then it will decommission the executors and if it does not need the node, it will free up the node.
78+
79+
***Note***:
80+
The maxExecutors will reserve the number of executors configured. Considering the example, even if you only use 2, it will reserve 6.
81+
82+
Hence, on enabling Dynamic allocation, Executors scale up or down based on the utilization of the Executors. This ensure that the Executors are provisioned in accordance with the needs of the job being run.
7883

7984
## Best practices
8085

0 commit comments

Comments
 (0)