Skip to content

Commit 563f4a7

Browse files
committed
[SPARK-53910] Add StatefulSet-based SparkApp example
### What changes were proposed in this pull request? This PR aims to add `StatefulSet`-based SparkApp example. ### Why are the changes needed? To provide a `StatefulSet` example with `spark.kubernetes.allocation.pods.allocator=statefulset`. Note that this requires the following is a part of `v0.6.0`. - #389 ### Does this PR introduce _any_ user-facing change? No, this is a new example. ### How was this patch tested? Manual review. ``` $ kubectl apply -f examples/pi-statefulset.yaml sparkapplication.spark.apache.org/pi-statefulset created ``` ``` $ kubectl get sparkapp NAME CURRENT STATE AGE pi-statefulset RunningHealthy 5s ``` ``` $ kubectl get pod NAME READY STATUS RESTARTS AGE pi-statefulset-0-driver 1/1 Running 0 7s spark-kubernetes-operator-cdb4b547d-5bhfm 1/1 Running 0 80s spark-s-pi-statefulset-0-0-0 1/1 Running 0 5s spark-s-pi-statefulset-0-0-1 1/1 Running 0 5s spark-s-pi-statefulset-0-0-2 1/1 Running 0 5s ``` ``` $ kubectl get sparkapp NAME CURRENT STATE AGE pi-statefulset ResourceReleased 48s ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #390 from dongjoon-hyun/SPARK-53910. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 55a1905 commit 563f4a7

File tree

1 file changed

+31
-0
lines changed

1 file changed

+31
-0
lines changed

examples/pi-statefulset.yaml

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
apiVersion: spark.apache.org/v1
16+
kind: SparkApplication
17+
metadata:
18+
name: pi-statefulset
19+
spec:
20+
mainClass: "org.apache.spark.examples.SparkPi"
21+
driverArgs: ["20000"]
22+
jars: "local:///opt/spark/examples/jars/spark-examples.jar"
23+
sparkConf:
24+
spark.executor.instances: "3"
25+
spark.kubernetes.allocation.pods.allocator: "statefulset"
26+
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
27+
spark.kubernetes.container.image: "apache/spark:4.0.1"
28+
applicationTolerations:
29+
resourceRetainPolicy: OnFailure
30+
runtimeVersions:
31+
sparkVersion: "4.0.1"

0 commit comments

Comments
 (0)