Skip to content

Commit 3b53d36

Browse files
ClifHouckClifHouck
authored
Merge pull request #37 from GeorgeJahad/addingSparkPi
Adding spark pi, (confirming driver and executor can talk to each other on Armada)
2 parents 7144daa + c4f9c64 commit 3b53d36

File tree

6 files changed

+123
-1
lines changed

6 files changed

+123
-1
lines changed

‎examples/runSparkPi.sh‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
#!/bin/bash
2+
3+
# Start up the driver, get it's ip address, then start the executor with it
4+
set -e
5+
6+
echo
7+
echo starting SparkPi driver
8+
armadactl submit examples/spark-pi-driver.yaml > /tmp/jobid.txt
9+
JOB_ID=`cat /tmp/jobid.txt | awk '{print $5}'`
10+
cat /tmp/jobid.txt
11+
echo
12+
13+
14+
echo waiting for SparkPi driver to start
15+
sleep 20
16+
17+
echo
18+
echo SparkPi driver ip addr:
19+
IP_ADDR=`kubectl get pod "armada-$JOB_ID-0" -o jsonpath='{.status.podIP}'`
20+
echo $IP_ADDR
21+
echo
22+
23+
echo passing drivers ip addr to executor and starting it
24+
IP_ADDR=$IP_ADDR envsubst < examples/spark-pi-executor.yaml > /tmp/ex.yaml
25+
armadactl submit /tmp/ex.yaml
26+
echo
27+
28+
echo SparkPi driver/executor started

‎examples/spark-pi-driver.yaml‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
queue: test
2+
jobSetId: job-set-1
3+
jobs:
4+
- namespace: default
5+
priority: 0
6+
podSpec:
7+
terminationGracePeriodSeconds: 0
8+
restartPolicy: Never
9+
containers:
10+
- name: spark-driver
11+
image: spark:testing
12+
env:
13+
- name: SPARK_DRIVER_BIND_ADDRESS
14+
valueFrom:
15+
fieldRef:
16+
apiVersion: v1
17+
fieldPath: status.podIP
18+
command:
19+
- /opt/entrypoint.sh
20+
args:
21+
- driver
22+
- --verbose
23+
- --class
24+
- org.apache.spark.examples.SparkPi
25+
- --master
26+
- armada://192.168.1.167:50051
27+
- --conf
28+
- "spark.driver.port=7078"
29+
- --conf
30+
- "spark.driver.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=0.0.0.0:5005"
31+
- local:///opt/spark/examples/jars/spark-examples.jar
32+
- "100"
33+
resources:
34+
limits:
35+
memory: 1Gi
36+
cpu: 1
37+
requests:
38+
memory: 1Gi
39+
cpu: 1
40+
ports:
41+
- containerPort: 7078
42+
name: driver-rpc-port
43+
protocol: TCP
44+
- containerPort: 7079
45+
name: blockmanager
46+
protocol: TCP
47+
- containerPort: 4040
48+
name: spark-ui
49+
protocol: TCP

‎examples/spark-pi-executor.yaml‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
queue: test
2+
jobSetId: job-set-1
3+
jobs:
4+
- namespace: default
5+
priority: 0
6+
podSpec:
7+
terminationGracePeriodSeconds: 0
8+
restartPolicy: Never
9+
containers:
10+
- name: spark-executor
11+
image: spark:testing
12+
env:
13+
- name: SPARK_EXECUTOR_MEMORY
14+
value: "512m"
15+
- name: SPARK_DRIVER_URL
16+
value: "spark://CoarseGrainedScheduler@${IP_ADDR}:7078"
17+
- name: SPARK_EXECUTOR_ID
18+
value: "1"
19+
- name: SPARK_EXECUTOR_CORES
20+
value: "1"
21+
- name: SPARK_APPLICATION_ID
22+
value: "test_spark_app_id"
23+
- name: SPARK_EXECUTOR_POD_IP
24+
valueFrom:
25+
fieldRef:
26+
apiVersion: v1
27+
fieldPath: status.podIP
28+
- name: SPARK_RESOURCE_PROFILE_ID
29+
value: "0"
30+
- name: SPARK_EXECUTOR_POD_NAME
31+
value: "test-pod-name"
32+
- name: SPARK_JAVA_OPT_0
33+
value: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
34+
command:
35+
- /opt/entrypoint.sh
36+
args:
37+
- executor
38+
resources:
39+
limits:
40+
memory: 1Gi
41+
cpu: 1
42+
requests:
43+
memory: 1Gi
44+
cpu: 1

resource-managers/armada/core/src/main/resources/META-INF/services/io.armadaproject.spark.deploy.SparkSubmitOperation renamed to resource-managers/armada/core/src/main/resources/META-INF/services/org.apache.spark.deploy.SparkSubmitOperation

File renamed without changes.

resource-managers/armada/core/src/main/resources/META-INF/services/io.armadaproject.spark.scheduler.ExternalClusterManager renamed to resource-managers/armada/core/src/main/resources/META-INF/services/org.apache.spark.scheduler.ExternalClusterManager

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,4 +15,4 @@
1515
# limitations under the License.
1616
#
1717

18-
org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager
18+
org.apache.spark.scheduler.cluster.armada.ArmadaClusterManager

‎resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,7 @@ case "$1" in
8585
"$SPARK_HOME/bin/spark-submit"
8686
--conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS"
8787
--conf "spark.executorEnv.SPARK_DRIVER_POD_IP=$SPARK_DRIVER_BIND_ADDRESS"
88+
--conf "spark.driver.host=$SPARK_DRIVER_BIND_ADDRESS"
8889
--deploy-mode client
8990
"$@"
9091
)

0 commit comments

Comments
 (0)