Skip to content

Commit 34ae166

Browse files
committed
[SPARK-52836] Fix sparkapp.sh to let Spark driver determine pod memory
### What changes were proposed in this pull request? This PR aims to fix the benchmark script `sparkapp.sh` to let Spark Driver determine pod request memory. ### Why are the changes needed? Apache Spark driver has `spark.driver.memory` is 1g by default and adds overhead when it creates pod. So, we had better use Spark's built-in logic. ### Does this PR introduce _any_ user-facing change? No, this is a benchmark script change. ### How was this patch tested? Manual run since the benchmark script is irrelevant to the CI. ``` $ cd tests/benchmark # The default value is 1k. But, we need to use small value on laptop. $ ./sparkapps.sh 50 CLEAN UP NAMESPACE FOR BENCHMARK START BENCHMARK WITH 50 JOBS FINISHED 50 JOBS IN 52 SECONDS. DELETED 50 JOBS IN 16 SECONDS. ``` ``` # While running the benchmark, we can check the memory. $ kubectl get pod -l spark-role=driver -oyaml | grep memory | sort | uniq -c 100 memory: 256Mi ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #289 from dongjoon-hyun/SPARK-52836. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent dc6a424 commit 34ae166

File tree

1 file changed

+2
-1
lines changed

1 file changed

+2
-1
lines changed

tests/benchmark/sparkapps.sh

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,9 @@ spec:
3838
jars: "local:///opt/spark/examples/jars/spark-examples.jar"
3939
driverArgs: ["0"]
4040
sparkConf:
41+
spark.driver.memory: "256m"
42+
spark.driver.memoryOverhead: "0m"
4143
spark.kubernetes.driver.request.cores: "100m"
42-
spark.kubernetes.driver.request.memory: "100Mi"
4344
spark.kubernetes.driver.master: "local[1]"
4445
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
4546
spark.kubernetes.container.image: "apache/spark:4.0.0-java21-scala"

0 commit comments

Comments
 (0)