Skip to content

Commit 9cf0bac

Browse files
committed
### What changes were proposed in this pull request? This PR aims to add an example to launch `Spark Connect Server` pod which uses the existing `Spark Cluster`. ### Why are the changes needed? Since `Spark Connect Server` pod is launched outside of `Spark Cluster`, we can expose the service port more easily. ### Does this PR introduce _any_ user-facing change? No. This is a new example. ### How was this patch tested? Manual testing. Launch `Spark K8s Operator`, `Spark Cluster`, and `Spark Connect Server` sequentially. ``` $ helm install spark-kubernetes-operator \ https://nightlies.apache.org/spark/charts/spark-kubernetes-operator-0.1.0-SNAPSHOT.tgz $ kubectl apply -f examples/prod-cluster-with-three-workers.yaml $ kubectl apply -f examples/spark-connect-server-with-spark-cluster.yaml ``` After exporting the `Spark Master` and `Spark Connect Server` ports, connect with `pyspark` client. ``` $ bin/pyspark --remote sc://localhost:15002 ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #169 from dongjoon-hyun/SPARK-51538. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 172c5b5 commit 9cf0bac

File tree

1 file changed

+30
-0
lines changed

1 file changed

+30
-0
lines changed
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
apiVersion: spark.apache.org/v1alpha1
16+
kind: SparkApplication
17+
metadata:
18+
name: spark-connect-server
19+
spec:
20+
mainClass: "org.apache.spark.sql.connect.service.SparkConnectServer"
21+
sparkConf:
22+
spark.master: "spark://prod-master-svc:7077"
23+
spark.submit.deployMode: "cluster"
24+
spark.executor.cores: "1"
25+
spark.cores.max: "3"
26+
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
27+
spark.kubernetes.container.image: "apache/spark:4.0.0-preview2"
28+
spark.ui.reverseProxy: "true"
29+
runtimeVersions:
30+
sparkVersion: "4.0.0-preview2"

0 commit comments

Comments
 (0)