Skip to content

Commit a6e837d

Browse files
committed
[SPARK-53663] Add spark-history-server-preview.yaml
### What changes were proposed in this pull request? Add `Spark History Server 4.1.0-preview` example (`spark-history-server-preview.yaml`) ### Why are the changes needed? To allow users to use `Apache Spark 4.1.0` history server easily. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ``` $ kubectl apply -f localstack.yml ``` ``` kubectl apply -f pi-with-eventlog.yaml ``` ``` $ kubectl apply -f spark-history-server-preview.yaml ``` <img width="509" height="381" alt="Screenshot 2025-09-22 at 11 20 46" src="https://github.com/user-attachments/assets/889d8378-4ccb-4753-ab15-0cc26ae7e9e7" /> ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#330 from dongjoon-hyun/SPARK-53663. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 991457c commit a6e837d

File tree

1 file changed

+43
-0
lines changed

1 file changed

+43
-0
lines changed
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
apiVersion: spark.apache.org/v1
16+
kind: SparkApplication
17+
metadata:
18+
name: spark-history-server
19+
spec:
20+
mainClass: "org.apache.spark.deploy.history.HistoryServer"
21+
sparkConf:
22+
spark.jars.packages: "org.apache.hadoop:hadoop-aws:3.4.1"
23+
spark.jars.ivy: "/tmp/.ivy2.5.2"
24+
spark.driver.memory: "2g"
25+
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
26+
spark.kubernetes.container.image: "apache/spark:4.1.0-preview1-java21-scala"
27+
spark.ui.port: "18080"
28+
spark.history.fs.logDirectory: "s3a://spark-events"
29+
spark.history.fs.cleaner.enabled: "true"
30+
spark.history.fs.cleaner.maxAge: "30d"
31+
spark.history.fs.cleaner.maxNum: "100"
32+
spark.history.fs.eventLog.rolling.maxFilesToRetain: "10"
33+
spark.hadoop.fs.defaultFS: "s3a://spark-events"
34+
spark.hadoop.fs.s3a.endpoint: "http://localstack:4566"
35+
spark.hadoop.fs.s3a.path.style.access: "true"
36+
spark.hadoop.fs.s3a.access.key: "test"
37+
spark.hadoop.fs.s3a.secret.key: "test"
38+
runtimeVersions:
39+
sparkVersion: "4.1.0-preview1"
40+
applicationTolerations:
41+
restartConfig:
42+
restartPolicy: Always
43+
maxRestartAttempts: 9223372036854775807

0 commit comments

Comments
 (0)