Skip to content

Commit 241592c

Browse files
committed
[SPARK-53932] Add FAIR schedule examples with Spark (Connect|Thrift) servers
### What changes were proposed in this pull request? This PR aims to add `FAIR` schedule examples by - Update `Spark 4.1.0-preview2`-based `Spark Connect Server` (4.1.0-preview2). - Add `Spark 4.1.0-preview2`-based `Spark Thrift Server` ### Why are the changes needed? To show `FAIR` schedule examples which is beneficial in the multi-tenant or concurrent job execution environments. <img width="566" height="267" alt="Screenshot 2025-10-15 at 19 52 31" src="https://github.com/user-attachments/assets/94968cc6-cee6-483d-8d57-924a0158a2b9" /> ### Does this PR introduce _any_ user-facing change? No behavior change because these are examples. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #393 from dongjoon-hyun/SPARK-53932. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 5543de8 commit 241592c

File tree

2 files changed

+39
-0
lines changed

2 files changed

+39
-0
lines changed

examples/spark-connect-server-preview.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,9 @@ spec:
2525
spark.dynamicAllocation.maxExecutors: "3"
2626
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
2727
spark.kubernetes.container.image: "apache/spark:{{SPARK_VERSION}}"
28+
spark.kubernetes.driver.pod.excludedFeatureSteps: "org.apache.spark.deploy.k8s.features.KerberosConfDriverFeatureStep"
29+
spark.kubernetes.executor.podNamePrefix: "spark-connect-server-preview"
30+
spark.scheduler.mode: "FAIR"
2831
applicationTolerations:
2932
resourceRetainPolicy: OnFailure
3033
runtimeVersions:
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
apiVersion: spark.apache.org/v1
16+
kind: SparkApplication
17+
metadata:
18+
name: spark-thrift-server-preview
19+
spec:
20+
mainClass: "org.apache.spark.sql.hive.thriftserver.HiveThriftServer2"
21+
sparkConf:
22+
spark.dynamicAllocation.enabled: "true"
23+
spark.dynamicAllocation.shuffleTracking.enabled: "true"
24+
spark.dynamicAllocation.minExecutors: "3"
25+
spark.dynamicAllocation.maxExecutors: "3"
26+
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
27+
spark.kubernetes.container.image: "apache/spark:{{SPARK_VERSION}}-java21-scala"
28+
spark.kubernetes.driver.pod.excludedFeatureSteps: "org.apache.spark.deploy.k8s.features.KerberosConfDriverFeatureStep"
29+
spark.kubernetes.executor.podNamePrefix: "spark-thrift-server-preview"
30+
spark.scheduler.mode: "FAIR"
31+
runtimeVersions:
32+
sparkVersion: "4.1.0-preview2"
33+
applicationTolerations:
34+
restartConfig:
35+
restartPolicy: Always
36+
maxRestartAttempts: 9223372036854775807

0 commit comments

Comments
 (0)