Skip to content

Commit d74c4d9

Browse files
committed
[SPARK-51382] Add submit-spark-connect-server.sh example
### What changes were proposed in this pull request? This PR aims to add `submit-spark-connect-server.sh` example to run `SparkConnectServer` on Spark Standalone Cluster. Note that there are other ways to launch `SparkConnectServer` and this only aims to focus on `Spark Standalone` case. ### Why are the changes needed? 1. Install Nightly `Spark Kubernetes Operator` and run a Spark cluster. ``` $ helm install spark-kubernetes-operator \ https://nightlies.apache.org/spark/charts/spark-kubernetes-operator-0.1.0-SNAPSHOT.tgz NAME: spark-kubernetes-operator LAST DEPLOYED: Mon Mar 3 20:55:41 2025 NAMESPACE: default STATUS: deployed REVISION: 1 $ kubectl apply -f prod-cluster-with-three-workers.yaml sparkcluster.spark.apache.org/prod created ``` 2. Setup port-forwarding. ``` $ kubectl port-forward svc/prod-master-svc 6066:6066 ``` ``` $ kubectl port-forward svc/prod-master-svc 6066:6066 ``` 3. Submit `SparkConnectServer`. ``` $ ./submit-spark-connect-server.sh { "action" : "CreateSubmissionResponse", "message" : "Driver successfully submitted as driver-20250304045839-0000", "serverSparkVersion" : "4.0.0-preview2", "submissionId" : "driver-20250304045839-0000", "success" : true }% ``` <img width="423" alt="Screenshot 2025-03-03 at 21 07 06" src="https://github.com/user-attachments/assets/e5e1f11b-86c6-4e4b-8cf9-02efc213c622" /> <img width="768" alt="Screenshot 2025-03-03 at 21 05 22" src="https://github.com/user-attachments/assets/013e92d4-a33b-40c0-a20d-d31827e0a18d" /> ### Does this PR introduce _any_ user-facing change? No. This is a newly added example. ### How was this patch tested? Manually follow the procedure. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#161 from dongjoon-hyun/SPARK-51382. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 563579f commit d74c4d9

File tree

1 file changed

+39
-0
lines changed

1 file changed

+39
-0
lines changed
Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
#!/usr/bin/env bash
2+
3+
#
4+
# Licensed to the Apache Software Foundation (ASF) under one or more
5+
# contributor license agreements. See the NOTICE file distributed with
6+
# this work for additional information regarding copyright ownership.
7+
# The ASF licenses this file to You under the Apache License, Version 2.0
8+
# (the "License"); you may not use this file except in compliance with
9+
# the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing, software
14+
# distributed under the License is distributed on an "AS IS" BASIS,
15+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16+
# See the License for the specific language governing permissions and
17+
# limitations under the License.
18+
#
19+
20+
curl -XPOST http://localhost:6066/v1/submissions/create \
21+
--data '{
22+
"appResource": "",
23+
"sparkProperties": {
24+
"spark.master": "spark://prod-master-svc:7077",
25+
"spark.submit.deployMode": "cluster",
26+
"spark.app.name": "SparkConnectServer",
27+
"spark.driver.cores": "1",
28+
"spark.driver.memory": "1g",
29+
"spark.executor.cores": "1",
30+
"spark.executor.memory": "1g",
31+
"spark.cores.max": "2",
32+
"spark.ui.reverseProxy": "true"
33+
},
34+
"clientSparkVersion": "",
35+
"mainClass": "org.apache.spark.sql.connect.service.SparkConnectServer",
36+
"environmentVariables": { },
37+
"action": "CreateSubmissionRequest",
38+
"appArgs": [ ]
39+
}'

0 commit comments

Comments
 (0)