[SPARK-49380] Add cluster-with-template.yaml example

### What changes were proposed in this pull request?

This PR aims to add `cluster-with-template.yaml` example.

### Why are the changes needed?

Apache Spark K8s Operator starts to support template-based Spark Cluster creation via the following.
- apache#80

We had better provide a concrete example like the following use cases.
- Custom annotations and labels.
- Custom full spec like `priorityClassName`, `securityContext`, `sidecar` containers

### Does this PR introduce _any_ user-facing change?

No. This is an unreleased new feature.

### How was this patch tested?

Manual review.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#101 from dongjoon-hyun/SPARK-49380.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>

Author: dongjoon-hyun

examples/cluster-with-template.yaml

@@ -0,0 +1,78 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+apiVersion: spark.apache.org/v1alpha1
+kind: SparkCluster
+metadata:
+  name: cluster-with-template
+spec:
+  runtimeVersions:
+    sparkVersion: "4.0.0-preview1"
+  clusterTolerations:
+    instanceConfig:
+      initWorkers: 1
+      minWorkers: 1
+      maxWorkers: 1
+  masterSpec:
+    masterStatefulSetMetadata:
+      annotations:
+        customAnnotation: "annotation"
+    masterStatefulSetSpec:
+      template:
+        spec:
+          priorityClassName: system-cluster-critical
+          securityContext:
+            runAsUser: 0
+          containers:
+          - name: sidecar
+            image: registry.k8s.io/pause
+            resources:
+              requests:
+                cpu: "0.1"
+                memory: "10Mi"
+              limits:
+                cpu: "0.1"
+                memory: "10Mi"
+    masterServiceMetadata:
+      annotations:
+        customAnnotation: "svc1"
+  workerSpec:
+    workerStatefulSetMetadata:
+      annotations:
+        customAnnotation: "annotation"
+    workerStatefulSetSpec:
+      template:
+        spec:
+          priorityClassName: system-cluster-critical
+          securityContext:
+            runAsUser: 0
+          containers:
+          - name: sidecar
+            image: registry.k8s.io/pause
+            resources:
+              requests:
+                cpu: "0.1"
+                memory: "10Mi"
+              limits:
+                cpu: "0.1"
+                memory: "10Mi"
+    workerServiceMetadata:
+      annotations:
+        customAnnotation: "annotation"
+  sparkConf:
+    spark.kubernetes.container.image: "spark:4.0.0-preview1"
+    spark.master.ui.title: "Spark Cluster with Template"
+    spark.master.rest.enabled: "true"
+    spark.master.rest.host: "0.0.0.0"
+    spark.ui.reverseProxy: "true"