Skip to content

Commit d171f17

Browse files
committed
[SPARK-54081] Add word-count-preview.yaml Example
### What changes were proposed in this pull request? This PR aims to add `word-count-preview.yaml` Example. ### Why are the changes needed? To provide a Spark Job shuffle example. <img width="396" height="212" alt="Screenshot 2025-10-29 at 21 14 08" src="https://github.com/user-attachments/assets/3945b010-7962-4c91-81f0-ed9199e65ab7" /> ### Does this PR introduce _any_ user-facing change? No behavior change because this is a new example. ### How was this patch tested? Manual review. ``` $ kubectl apply -f examples/word-count-preview.yaml ``` ``` $ kubectl logs -f word-count-0-driver ... 25/10/29 23:48:04 INFO DAGScheduler: Job 0 finished: collect at JavaWordCount.java:53, took 3544.885793 ms -Psparkr: 1 -B: 1 Spark: 1 -Pkubernetes: 1 -Pyarn: 1 revision: 1 Build: 1 built: 1 c5ff48cc2b2): 1 (git: 1 -Phadoop-3: 1 4.1.0-preview2: 1 flags:: 1 -Phive-thriftserver: 1 for: 1 -Phive: 1 3.4.2: 1 Hadoop: 1 ``` ``` $ k get sparkapp NAME CURRENT STATE AGE word-count ResourceReleased 2m35s ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #408 from dongjoon-hyun/SPARK-54081. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 45cd118 commit d171f17

File tree

1 file changed

+32
-0
lines changed

1 file changed

+32
-0
lines changed

examples/word-count-preview.yaml

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
apiVersion: spark.apache.org/v1
16+
kind: SparkApplication
17+
metadata:
18+
name: word-count
19+
spec:
20+
mainClass: "org.apache.spark.examples.JavaWordCount"
21+
driverArgs: [ "/opt/spark/RELEASE" ]
22+
jars: "local:///opt/spark/examples/jars/spark-examples.jar"
23+
sparkConf:
24+
spark.dynamicAllocation.enabled: "true"
25+
spark.dynamicAllocation.maxExecutors: "3"
26+
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
27+
spark.kubernetes.container.image: "apache/spark:{{SPARK_VERSION}}-java21-scala"
28+
spark.kubernetes.driver.pod.excludedFeatureSteps: "org.apache.spark.deploy.k8s.features.KerberosConfDriverFeatureStep"
29+
applicationTolerations:
30+
resourceRetainPolicy: OnFailure
31+
runtimeVersions:
32+
sparkVersion: "4.1.0-preview2"

0 commit comments

Comments
 (0)