Skip to content

Commit aee213f

Browse files
dongjoon-hyundongjoon-hyun
committed
[SPARK-52074] Add Dockerfiles to all examples
### What changes were proposed in this pull request? This PR aims to add `Dockerfile`s to all examples. Also, all example `README.md` files are updated. ### Why are the changes needed? To provide an easy way to build and use with Docker and K8s environment. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #132 from dongjoon-hyun/SPARK-52074. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent a2c8e0d commit aee213f

File tree

11 files changed

+314
-20
lines changed

11 files changed

+314
-20
lines changed

‎Examples/app/Dockerfile‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
FROM swift:6.1 AS builder
18+
19+
WORKDIR /app
20+
21+
COPY . .
22+
23+
RUN swift build -c release
24+
25+
FROM swift:6.1-slim
26+
27+
ARG SPARK_UID=185
28+
29+
LABEL org.opencontainers.image.authors="Apache Spark project <[email protected]>"
30+
LABEL org.opencontainers.image.licenses="Apache-2.0"
31+
LABEL org.opencontainers.image.ref.name="Apache Spark Connect for Swift"
32+
33+
ENV SPARK_SWIFT_HOME=/opt/spark-swift
34+
ENV SPARK_SWIFT_APP=SparkConnectSwiftApp
35+
36+
WORKDIR $SPARK_SWIFT_HOME
37+
38+
RUN groupadd --system --gid=$SPARK_UID spark && \
39+
useradd --system --home-dir $SPARK_SWIFT_HOME --uid=$SPARK_UID --gid=spark spark && \
40+
chown -R spark:spark $SPARK_SWIFT_HOME
41+
42+
COPY --from=builder --chown=spark:spark /app/.build/*-unknown-linux-gnu/release/$SPARK_SWIFT_APP .
43+
44+
USER spark
45+
46+
ENTRYPOINT ["/bin/sh", "-c", "$SPARK_SWIFT_HOME/$SPARK_SWIFT_APP"]

‎Examples/app/README.md‎

Lines changed: 45 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,55 @@ This is an example Swift application to show how to use Apache Spark Connect Swi
44

55
## How to run
66

7-
Run this Swift application.
7+
Prepare `Spark Connect Server` via running Docker image.
8+
9+
```
10+
docker run --rm -p 15002:15002 apache/spark:4.0.0-preview2 bash -c "/opt/spark/sbin/start-connect-server.sh --wait"
11+
```
12+
13+
Build an application Docker image.
14+
15+
```
16+
$ docker build -t apache/spark-connect-swift:app .
17+
$ docker images apache/spark-connect-swift:app
18+
REPOSITORY TAG IMAGE ID CREATED SIZE
19+
apache/spark-connect-swift app e132e1b38348 5 seconds ago 368MB
20+
```
21+
22+
Run `app` docker image.
23+
24+
```
25+
$ docker run --rm -e SPARK_REMOTE=sc://host.docker.internal:15002 apache/spark-connect-swift:app
26+
Connected to Apache Spark 4.0.0-preview2 Server
27+
EXECUTE: DROP TABLE IF EXISTS t
28+
EXECUTE: CREATE TABLE IF NOT EXISTS t(a INT) USING ORC
29+
EXECUTE: INSERT INTO t VALUES (1), (2), (3)
30+
SELECT * FROM t
31+
+---+
32+
| a|
33+
+---+
34+
| 2|
35+
| 1|
36+
| 3|
37+
+---+
38+
39+
+---+
40+
| id|
41+
+---+
42+
| 0|
43+
| 8|
44+
| 6|
45+
| 2|
46+
| 4|
47+
+---+
48+
```
49+
50+
Run from source code.
851

952
```
1053
$ swift run
1154
...
12-
Connected to Apache Spark 4.0.0 Server
55+
Connected to Apache Spark 4.0.0-preview2 Server
1356
EXECUTE: DROP TABLE IF EXISTS t
1457
EXECUTE: CREATE TABLE IF NOT EXISTS t(a INT) USING ORC
1558
EXECUTE: INSERT INTO t VALUES (1), (2), (3)

‎Examples/pi/Dockerfile‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
FROM swift:6.1 AS builder
18+
19+
WORKDIR /app
20+
21+
COPY . .
22+
23+
RUN swift build -c release
24+
25+
FROM swift:6.1-slim
26+
27+
ARG SPARK_UID=185
28+
29+
LABEL org.opencontainers.image.authors="Apache Spark project <[email protected]>"
30+
LABEL org.opencontainers.image.licenses="Apache-2.0"
31+
LABEL org.opencontainers.image.ref.name="Apache Spark Connect for Swift"
32+
33+
ENV SPARK_SWIFT_HOME=/opt/spark-swift
34+
ENV SPARK_SWIFT_APP=SparkConnectSwiftPi
35+
36+
WORKDIR $SPARK_SWIFT_HOME
37+
38+
RUN groupadd --system --gid=$SPARK_UID spark && \
39+
useradd --system --home-dir $SPARK_SWIFT_HOME --uid=$SPARK_UID --gid=spark spark && \
40+
chown -R spark:spark $SPARK_SWIFT_HOME
41+
42+
COPY --from=builder --chown=spark:spark /app/.build/*-unknown-linux-gnu/release/$SPARK_SWIFT_APP .
43+
44+
USER spark
45+
46+
ENTRYPOINT ["/bin/sh", "-c", "$SPARK_SWIFT_HOME/$SPARK_SWIFT_APP"]

‎Examples/pi/README.md‎

Lines changed: 23 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,31 @@ This is an example Swift application to show how to use Apache Spark Connect Swi
44

55
## How to run
66

7-
Run this Swift application.
7+
Prepare `Spark Connect Server` via running Docker image.
8+
```
9+
docker run --rm -p 15002:15002 apache/spark:4.0.0-preview2 bash -c "/opt/spark/sbin/start-connect-server.sh --wait"
10+
```
11+
12+
Build an application Docker image.
13+
14+
```
15+
$ docker build -t apache/spark-connect-swift:pi .
16+
$ docker images apache/spark-connect-swift:pi
17+
REPOSITORY TAG IMAGE ID CREATED SIZE
18+
apache/spark-connect-swift pi d03952577564 4 seconds ago 369MB
19+
```
20+
21+
Run `pi` docker image.
22+
23+
```
24+
$ docker run --rm -e SPARK_REMOTE=sc://host.docker.internal:15002 apache/spark-connect-swift:pi
25+
Pi is roughly 3.1412831412831412
26+
```
27+
28+
Run from source code.
829

930
```
10-
$ swift run SparkConnectSwiftPi 1000000
31+
$ swift run
1132
...
12-
Connected to Apache Spark 4.0.0 Server
1333
Pi is roughly 3.1423711423711422
1434
```

‎Examples/stream/Dockerfile‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
FROM swift:6.1 AS builder
18+
19+
WORKDIR /app
20+
21+
COPY . .
22+
23+
RUN swift build -c release
24+
25+
FROM swift:6.1-slim
26+
27+
ARG SPARK_UID=185
28+
29+
LABEL org.opencontainers.image.authors="Apache Spark project <[email protected]>"
30+
LABEL org.opencontainers.image.licenses="Apache-2.0"
31+
LABEL org.opencontainers.image.ref.name="Apache Spark Connect for Swift"
32+
33+
ENV SPARK_SWIFT_HOME=/opt/spark-swift
34+
ENV SPARK_SWIFT_APP=SparkConnectSwiftNetworkWordCount
35+
36+
WORKDIR $SPARK_SWIFT_HOME
37+
38+
RUN groupadd --system --gid=$SPARK_UID spark && \
39+
useradd --system --home-dir $SPARK_SWIFT_HOME --uid=$SPARK_UID --gid=spark spark && \
40+
chown -R spark:spark $SPARK_SWIFT_HOME
41+
42+
COPY --from=builder --chown=spark:spark /app/.build/*-unknown-linux-gnu/release/$SPARK_SWIFT_APP .
43+
44+
USER spark
45+
46+
ENTRYPOINT ["/bin/sh", "-c", "$SPARK_SWIFT_HOME/$SPARK_SWIFT_APP"]

‎Examples/stream/README.md‎

Lines changed: 24 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ This is an example Swift stream processing application to show how to count word
55
## Run `Spark Connect Server`
66

77
```bash
8-
./sbin/start-connect-server.sh --wait -c spark.log.level=ERROR
8+
docker run --rm -p 15002:15002 apache/spark:4.0.0-preview2 bash -c "/opt/spark/sbin/start-connect-server.sh --wait -c spark.log.level=ERROR"
99
```
1010

1111
## Run `Netcat` as a streaming input server
@@ -16,12 +16,21 @@ You will first need to run Netcat (a small utility found in most Unix-like syste
1616
nc -lk 9999
1717
```
1818

19-
## Start streaming processing application
19+
## Build and run from docker image
20+
21+
Build an application Docker image.
2022

21-
```bash
22-
$ swift run
23-
...
24-
Connected to Apache Spark 4.0.0 Server
23+
```
24+
$ docker build -t apache/spark-connect-swift:stream .
25+
$ docker images apache/spark-connect-swift:stream
26+
REPOSITORY TAG IMAGE ID CREATED SIZE
27+
apache/spark-connect-swift stream a4daa10ad9c5 7 seconds ago 369MB
28+
```
29+
30+
Run `stream` docker image.
31+
32+
```
33+
$ docker run --rm -e SPARK_REMOTE=sc://host.docker.internal:15002 -e TARGET_HOST=host.docker.internal apache/spark-connect-swift:stream
2534
```
2635

2736
## Send input and check output
@@ -34,7 +43,7 @@ apache spark
3443
apache hadoop
3544
```
3645

37-
`Spark Connect Server` output will look something like the following.
46+
`Spark Connect Server` output will look something like the following.
3847

3948
```bash
4049
-------------------------------------------
@@ -66,3 +75,11 @@ Batch: 2
6675
|hadoop| 1|
6776
+------+--------+
6877
```
78+
79+
## Run from source code
80+
81+
```bash
82+
$ swift run
83+
...
84+
Connected to Apache Spark 4.0.0 Server
85+
```

‎Examples/stream/Sources/main.swift‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,17 +17,20 @@
1717
// under the License.
1818
//
1919

20+
import Foundation
2021
import SparkConnect
2122

2223
let spark = try await SparkSession.builder.getOrCreate()
2324
print("Connected to Apache Spark \(await spark.version) Server")
2425

26+
let host = ProcessInfo.processInfo.environment["TARGET_HOST"] ?? "localhost"
27+
2528
let lines =
2629
await spark
2730
.readStream
2831
.format("socket")
29-
.option("host", "localhost")
30-
.option("port", "9999")
32+
.option("host", host)
33+
.option("port", 9999)
3134
.load()
3235

3336
let word =

‎Examples/web/Dockerfile‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
FROM swift:6.1 AS builder
18+
19+
WORKDIR /app
20+
21+
COPY . .
22+
23+
RUN swift build -c release
24+
25+
FROM swift:6.1-slim
26+
27+
ARG SPARK_UID=185
28+
29+
LABEL org.opencontainers.image.authors="Apache Spark project <[email protected]>"
30+
LABEL org.opencontainers.image.licenses="Apache-2.0"
31+
LABEL org.opencontainers.image.ref.name="Apache Spark Connect for Swift"
32+
33+
ENV SPARK_SWIFT_HOME=/opt/spark-swift
34+
ENV SPARK_SWIFT_APP=SparkConnectSwiftWeb
35+
36+
WORKDIR $SPARK_SWIFT_HOME
37+
38+
RUN groupadd --system --gid=$SPARK_UID spark && \
39+
useradd --system --home-dir $SPARK_SWIFT_HOME --uid=$SPARK_UID --gid=spark spark && \
40+
chown -R spark:spark $SPARK_SWIFT_HOME
41+
42+
COPY --from=builder --chown=spark:spark /app/.build/*-unknown-linux-gnu/release/$SPARK_SWIFT_APP .
43+
44+
USER spark
45+
46+
EXPOSE 8080
47+
48+
ENTRYPOINT ["/bin/sh", "-c", "$SPARK_SWIFT_HOME/$SPARK_SWIFT_APP"]

‎Examples/web/Package.swift‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@
2020
import PackageDescription
2121

2222
let package = Package(
23-
name: "SparkConnectSwiftWebapp",
23+
name: "SparkConnectSwiftWeb",
2424
platforms: [
2525
.macOS(.v15)
2626
],
@@ -33,7 +33,7 @@ let package = Package(
3333
],
3434
targets: [
3535
.executableTarget(
36-
name: "SparkConnectSwiftWebapp",
36+
name: "SparkConnectSwiftWeb",
3737
dependencies: [
3838
.product(name: "Vapor", package: "vapor"),
3939
.product(name: "NIOCore", package: "swift-nio"),

0 commit comments

Comments
 (0)