Skip to content

Commit 37bf26b

Browse files
committed
[SPARK-53994] Exclude KerberosConfDriverFeatureStep during benchmarking
### What changes were proposed in this pull request? This PR aims to exclude `KerberosConfDriverFeatureStep` during benchmarking because this is not used during `SparkApp` submission benchmark. ### Why are the changes needed? To suppress irrelevant warning messages from Hadoop layer. ### Does this PR introduce _any_ user-facing change? No. This is a test script change. ### How was this patch tested? Manual review. ``` $ ./tests/benchmark/sparkapps.sh CLEAN UP NAMESPACE FOR BENCHMARK No resources found START BENCHMARK WITH 1000 JOBS FINISHED 1000 JOBS IN 193 SECONDS. DELETED 1000 JOBS IN 254 SECONDS. ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #404 from dongjoon-hyun/SPARK-53994. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 4497f07 commit 37bf26b

File tree

1 file changed

+3
-2
lines changed

1 file changed

+3
-2
lines changed

tests/benchmark/sparkapps.sh

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -40,10 +40,11 @@ spec:
4040
sparkConf:
4141
spark.driver.memory: "256m"
4242
spark.driver.memoryOverhead: "0m"
43-
spark.kubernetes.driver.request.cores: "100m"
44-
spark.kubernetes.driver.master: "local[1]"
4543
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
4644
spark.kubernetes.container.image: "apache/spark:4.0.1-java21-scala"
45+
spark.kubernetes.driver.master: "local[1]"
46+
spark.kubernetes.driver.pod.excludedFeatureSteps: "org.apache.spark.deploy.k8s.features.KerberosConfDriverFeatureStep"
47+
spark.kubernetes.driver.request.cores: "100m"
4748
runtimeVersions:
4849
sparkVersion: "4.0.1"
4950
---

0 commit comments

Comments
 (0)