Misc fix

* introduce the new field to both v1alpha1 and v1beta1
* remove unrelated typo fix
* remove scala version from new e2e
* update image for new e2e scenario
* style fix

Author: jiangzho

build-tools/helm/spark-kubernetes-operator/crds/sparkapplications.spark.apache.org-v1.yaml

@@ -57,6 +57,8 @@ spec:
                         minExecutors:
                           type: integer
                       type: object
+                    resourceRetainDurationMillis:
+                      type: integer
                     resourceRetainPolicy:
                       enum:
                         - Always
@@ -8734,6 +8736,8 @@ spec:
                         minExecutors:
                           type: integer
                       type: object
+                    resourceRetainDurationMillis:
+                      type: integer
                     resourceRetainPolicy:
                       enum:
                         - Always

docs/spark_custom_resources.md

@@ -290,14 +290,13 @@ On the other hand, when developing an application, it's possible to configure
 applicationTolerations:
   # Acceptable values are 'Always', 'OnFailure', 'Never'
   # Setting this to 'OnFailure' would retain secondary resources if and only if the app fails
-  resourceRetainPolicy: OnFailure
+  resourceRetentionPolicy: OnFailure
   # Secondary resources would be garbage collected 10 minutes after app termination 
   resourceRetainDurationMillis: 600000
-
 ```
 
 to avoid operator attempt to delete driver pod and driver resources if app fails. Similarly,
-if resourceRetainPolicy is set to `Always`, operator would not delete driver resources
+if resourceRetentionPolicy is set to `Always`, operator would not delete driver resources
 when app ends. They would be by default kept with the same lifecycle as the App. It's also
 possible to configure `resourceRetainDurationMillis` to define the maximal retain duration for
 these resources. Note that this applies only to operator-created resources (driver pod, SparkConf

spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/AppCleanUpStep.java

@@ -75,7 +75,7 @@ public AppCleanUpStep(Supplier<ApplicationState> onDemandCleanUpReason) {
    *   <li>When the application is being deleted on demand(e.g. being deleted) with a reason
    *   <li>When the application is stopping
    *   <li>When the application has terminated without releasing resources, but it has exceeded
-   *       configured retain duration
+   *       configured retention duration
    * </ul>
    *
    * <p>It would proceed to next steps with no actions for application in other states. Note that

tests/e2e/resource-retain-duration/chainsaw-test.yaml

@@ -39,7 +39,7 @@ spec:
         bindings:
           - name: SPARK_APP_NAMESPACE
             value: default
-        timeout: 60s
+        timeout: 120s
         file: "../assertions/spark-application/spark-state-transition-with-retain-check.yaml"
     catch:
     - describe:
@@ -51,6 +51,6 @@ spec:
         env:
           - name: SPARK_APPLICATION_NAME
             value: ($SPARK_APPLICATION_NAME)
-        timeout: 60s
+        timeout: 120s
         content: |
           kubectl delete sparkapplication $SPARK_APPLICATION_NAME

tests/e2e/resource-retain-duration/spark-example-retain-duration.yaml

@@ -28,8 +28,7 @@ spec:
     resourceRetainDurationMillis: 10000
   sparkConf:
     spark.executor.instances: "1"
-    spark.kubernetes.container.image: "apache/spark:4.0.0-scala2.13-java17-ubuntu"
+    spark.kubernetes.container.image: "apache/spark:4.0.0-java21-scala"
     spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
   runtimeVersions:
     sparkVersion: 4.0.0
-    scalaVersion: "2.13"