Skip to content

Commit 4563830

Browse files
committed
[SPARK-53852] Add spark.kubernetes.driver.pod.excludedFeatureSteps usage example
### What changes were proposed in this pull request? This PR aims to add `spark.kubernetes.driver.pod.excludedFeatureSteps` usage example. ### Why are the changes needed? Since `Apache Spark K8s Operator v0.6.0`, we can use new features from `Apache Spark 4.1.0-preview2`. - apache#364 For example, `spark.kubernetes.driver.pod.excludedFeatureSteps` allows us to skip the specific feature steps. If we set the following, we can skip `KerberosConfDriverFeatureStep` which fails always on Java 25 environment currently. - apache/spark#51522 ``` spark.kubernetes.driver.pod.excludedFeatureSteps: "org.apache.spark.deploy.k8s.features.KerberosConfDriverFeatureStep" ``` **BEFORE** ``` 25/10/09 04:22:23 INFO pi-preview default o.a.s.i.Logging You have not specified a krb5.conf file locally or via a ConfigMap. Make sure that you have the krb5.conf locally on the driver image. WARNING: A restricted method in java.lang.System has been called WARNING: java.lang.System::loadLibrary has been called by org.apache.hadoop.util.NativeCodeLoader in an unnamed module (file:/opt/spark-operator/operator/spark-kubernetes-operator.jar) WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module WARNING: Restricted methods will be blocked in a future release unless native access is enabled 25/10/09 04:22:23 WARN pi-preview default o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 25/10/09 04:22:23 WARN pi-preview default o.a.s.i.Logging Fail to get credentials java.lang.UnsupportedOperationException: getSubject is not supported at java.base/javax.security.auth.Subject.getSubject(Subject.java:277) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:588) at org.apache.spark.deploy.k8s.features.KerberosConfDriverFeatureStep.liftedTree1$1(KerberosConfDriverFeatureStep.scala:95) ``` **AFTER** ``` 25/10/09 04:20:34 INFO pi-preview default o.a.s.i.Logging You have not specified a krb5.conf file locally or via a ConfigMap. Make sure that you have the krb5.conf locally on the driver image. ``` ### Does this PR introduce _any_ user-facing change? No. This is an example update. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#379 from dongjoon-hyun/SPARK-53852. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 5a2de5b commit 4563830

File tree

2 files changed

+2
-0
lines changed

2 files changed

+2
-0
lines changed

examples/pi-preview.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ spec:
2525
spark.dynamicAllocation.maxExecutors: "3"
2626
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
2727
spark.kubernetes.container.image: "apache/spark:4.1.0-preview2-java21-scala"
28+
spark.kubernetes.driver.pod.excludedFeatureSteps: "org.apache.spark.deploy.k8s.features.KerberosConfDriverFeatureStep"
2829
applicationTolerations:
2930
resourceRetainPolicy: OnFailure
3031
runtimeVersions:

examples/pi.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ spec:
2525
spark.dynamicAllocation.maxExecutors: "3"
2626
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
2727
spark.kubernetes.container.image: "apache/spark:4.0.1"
28+
spark.kubernetes.driver.pod.excludedFeatureSteps: "org.apache.spark.deploy.k8s.features.KerberosConfDriverFeatureStep"
2829
applicationTolerations:
2930
resourceRetainPolicy: OnFailure
3031
runtimeVersions:

0 commit comments

Comments
 (0)