Commit 34ae166
committed
[SPARK-52836] Fix
### What changes were proposed in this pull request?
This PR aims to fix the benchmark script `sparkapp.sh` to let Spark Driver determine pod request memory.
### Why are the changes needed?
Apache Spark driver has `spark.driver.memory` is 1g by default and adds overhead when it creates pod. So, we had better use Spark's built-in logic.
### Does this PR introduce _any_ user-facing change?
No, this is a benchmark script change.
### How was this patch tested?
Manual run since the benchmark script is irrelevant to the CI.
```
$ cd tests/benchmark
# The default value is 1k. But, we need to use small value on laptop.
$ ./sparkapps.sh 50
CLEAN UP NAMESPACE FOR BENCHMARK
START BENCHMARK WITH 50 JOBS
FINISHED 50 JOBS IN 52 SECONDS.
DELETED 50 JOBS IN 16 SECONDS.
```
```
# While running the benchmark, we can check the memory.
$ kubectl get pod -l spark-role=driver -oyaml | grep memory | sort | uniq -c
100 memory: 256Mi
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #289 from dongjoon-hyun/SPARK-52836.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>sparkapp.sh to let Spark driver determine pod memory1 parent dc6a424 commit 34ae166
1 file changed
+2
-1
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| 41 | + | |
| 42 | + | |
41 | 43 | | |
42 | | - | |
43 | 44 | | |
44 | 45 | | |
45 | 46 | | |
| |||
0 commit comments