Commit 4563830
committed
[SPARK-53852] Add
### What changes were proposed in this pull request?
This PR aims to add `spark.kubernetes.driver.pod.excludedFeatureSteps` usage example.
### Why are the changes needed?
Since `Apache Spark K8s Operator v0.6.0`, we can use new features from `Apache Spark 4.1.0-preview2`.
- #364
For example, `spark.kubernetes.driver.pod.excludedFeatureSteps` allows us to skip the specific feature steps. If we set the following, we can skip `KerberosConfDriverFeatureStep` which fails always on Java 25 environment currently.
- apache/spark#51522
```
spark.kubernetes.driver.pod.excludedFeatureSteps: "org.apache.spark.deploy.k8s.features.KerberosConfDriverFeatureStep"
```
**BEFORE**
```
25/10/09 04:22:23 INFO pi-preview default o.a.s.i.Logging You have not specified a krb5.conf file locally or via a ConfigMap. Make sure that you have the krb5.conf locally on the driver image.
WARNING: A restricted method in java.lang.System has been called
WARNING: java.lang.System::loadLibrary has been called by org.apache.hadoop.util.NativeCodeLoader in an unnamed module (file:/opt/spark-operator/operator/spark-kubernetes-operator.jar)
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module
WARNING: Restricted methods will be blocked in a future release unless native access is enabled
25/10/09 04:22:23 WARN pi-preview default o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
25/10/09 04:22:23 WARN pi-preview default o.a.s.i.Logging Fail to get credentials
java.lang.UnsupportedOperationException: getSubject is not supported
at java.base/javax.security.auth.Subject.getSubject(Subject.java:277)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:588)
at org.apache.spark.deploy.k8s.features.KerberosConfDriverFeatureStep.liftedTree1$1(KerberosConfDriverFeatureStep.scala:95)
```
**AFTER**
```
25/10/09 04:20:34 INFO pi-preview default o.a.s.i.Logging You have not specified a krb5.conf file locally or via a ConfigMap. Make sure that you have the krb5.conf locally on the driver image.
```
### Does this PR introduce _any_ user-facing change?
No. This is an example update.
### How was this patch tested?
Manual review.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #379 from dongjoon-hyun/SPARK-53852.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>spark.kubernetes.driver.pod.excludedFeatureSteps usage example1 parent 5a2de5b commit 4563830
2 files changed
+2
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| 28 | + | |
28 | 29 | | |
29 | 30 | | |
30 | 31 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| 28 | + | |
28 | 29 | | |
29 | 30 | | |
30 | 31 | | |
| |||
0 commit comments