Skip to content

Commit a9b267e

Browse files
dongjoon-hyundongjoon-hyun
committed
[SPARK-52067] Add Examples directory
### What changes were proposed in this pull request? This PR aims to add `Examples` directory. ### Why are the changes needed? To provide ASF-licensed examples officially. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#124 from dongjoon-hyun/SPARK-52067. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 5ded181 commit a9b267e

File tree

9 files changed

+369
-4
lines changed

9 files changed

+369
-4
lines changed

‎Examples/app/Package.swift‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
// swift-tools-version: 6.0
2+
//
3+
// Licensed to the Apache Software Foundation (ASF) under one
4+
// or more contributor license agreements. See the NOTICE file
5+
// distributed with this work for additional information
6+
// regarding copyright ownership. The ASF licenses this file
7+
// to you under the Apache License, Version 2.0 (the
8+
// "License"); you may not use this file except in compliance
9+
// with the License. You may obtain a copy of the License at
10+
//
11+
// http://www.apache.org/licenses/LICENSE-2.0
12+
//
13+
// Unless required by applicable law or agreed to in writing,
14+
// software distributed under the License is distributed on an
15+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
// KIND, either express or implied. See the License for the
17+
// specific language governing permissions and limitations
18+
// under the License.
19+
//
20+
21+
import PackageDescription
22+
23+
let package = Package(
24+
name: "SparkConnectSwiftApp",
25+
platforms: [
26+
.macOS(.v15)
27+
],
28+
dependencies: [
29+
.package(url: "https://github.com/apache/spark-connect-swift.git", branch: "v0.1.0")
30+
],
31+
targets: [
32+
.executableTarget(
33+
name: "SparkConnectSwiftApp",
34+
dependencies: [.product(name: "SparkConnect", package: "spark-connect-swift")]
35+
)
36+
]
37+
)

‎Examples/app/README.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# A Swift Application with Apache Spark Connect Swift Client
2+
3+
This is an example Swift application to show how to use Apache Spark Connect Swift Client library.
4+
5+
## How to run
6+
7+
Run this Swift application.
8+
9+
```
10+
$ swift run
11+
...
12+
Connected to Apache Spark 4.0.0 Server
13+
EXECUTE: DROP TABLE IF EXISTS t
14+
EXECUTE: CREATE TABLE IF NOT EXISTS t(a INT) USING ORC
15+
EXECUTE: INSERT INTO t VALUES (1), (2), (3)
16+
SELECT * FROM t
17+
+---+
18+
| a |
19+
+---+
20+
| 2 |
21+
| 1 |
22+
| 3 |
23+
+---+
24+
+----+
25+
| id |
26+
+----+
27+
| 2 |
28+
| 6 |
29+
| 0 |
30+
| 8 |
31+
| 4 |
32+
+----+
33+
```

‎Examples/app/Sources/main.swift‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
//
2+
// Licensed to the Apache Software Foundation (ASF) under one
3+
// or more contributor license agreements. See the NOTICE file
4+
// distributed with this work for additional information
5+
// regarding copyright ownership. The ASF licenses this file
6+
// to you under the Apache License, Version 2.0 (the
7+
// "License"); you may not use this file except in compliance
8+
// with the License. You may obtain a copy of the License at
9+
//
10+
// http://www.apache.org/licenses/LICENSE-2.0
11+
//
12+
// Unless required by applicable law or agreed to in writing,
13+
// software distributed under the License is distributed on an
14+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
// KIND, either express or implied. See the License for the
16+
// specific language governing permissions and limitations
17+
// under the License.
18+
//
19+
20+
import SparkConnect
21+
22+
let spark = try await SparkSession.builder.getOrCreate()
23+
print("Connected to Apache Spark \(await spark.version) Server")
24+
25+
let statements = [
26+
"DROP TABLE IF EXISTS t",
27+
"CREATE TABLE IF NOT EXISTS t(a INT) USING ORC",
28+
"INSERT INTO t VALUES (1), (2), (3)",
29+
]
30+
31+
for s in statements {
32+
print("EXECUTE: \(s)")
33+
_ = try await spark.sql(s).count()
34+
}
35+
print("SELECT * FROM t")
36+
try await spark.sql("SELECT * FROM t").cache().show()
37+
38+
try await spark.range(10).filter("id % 2 == 0").write.mode("overwrite").orc("/tmp/orc")
39+
try await spark.read.orc("/tmp/orc").show()
40+
41+
await spark.stop()

‎Examples/web/Package.swift‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
// swift-tools-version:6.0
2+
//
3+
// Licensed to the Apache Software Foundation (ASF) under one
4+
// or more contributor license agreements. See the NOTICE file
5+
// distributed with this work for additional information
6+
// regarding copyright ownership. The ASF licenses this file
7+
// to you under the Apache License, Version 2.0 (the
8+
// "License"); you may not use this file except in compliance
9+
// with the License. You may obtain a copy of the License at
10+
//
11+
// http://www.apache.org/licenses/LICENSE-2.0
12+
//
13+
// Unless required by applicable law or agreed to in writing,
14+
// software distributed under the License is distributed on an
15+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
// KIND, either express or implied. See the License for the
17+
// specific language governing permissions and limitations
18+
// under the License.
19+
//
20+
import PackageDescription
21+
22+
let package = Package(
23+
name: "SparkConnectSwiftWebapp",
24+
platforms: [
25+
.macOS(.v15)
26+
],
27+
dependencies: [
28+
// 💧 A server-side Swift web framework.
29+
.package(url: "https://github.com/vapor/vapor.git", from: "4.110.1"),
30+
// 🔵 Non-blocking, event-driven networking for Swift. Used for custom executors
31+
.package(url: "https://github.com/apple/swift-nio.git", from: "2.65.0"),
32+
.package(url: "https://github.com/apache/spark-connect-swift.git", branch: "v0.1.0"),
33+
],
34+
targets: [
35+
.executableTarget(
36+
name: "SparkConnectSwiftWebapp",
37+
dependencies: [
38+
.product(name: "Vapor", package: "vapor"),
39+
.product(name: "NIOCore", package: "swift-nio"),
40+
.product(name: "NIOPosix", package: "swift-nio"),
41+
.product(name: "SparkConnect", package: "spark-connect-swift"),
42+
],
43+
swiftSettings: swiftSettings
44+
)
45+
]
46+
)
47+
48+
var swiftSettings: [SwiftSetting] {
49+
[
50+
.enableUpcomingFeature("ExistentialAny")
51+
]
52+
}

‎Examples/web/README.md‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# A Swift Application with Apache Spark Connect Swift Client
2+
3+
This project is designed to illustrate a Swift-based HTTP WebServer with Apache Spark Connect.
4+
5+
- https://swiftpackageindex.com/apache/spark-connect-swift
6+
- https://swiftpackageindex.com/vapor/vapor
7+
8+
## Create a Swift project
9+
10+
```
11+
brew install vapor
12+
vapor new spark-connect-swift-web -n
13+
```
14+
15+
## Use `Apache Spark Connect Swift Client` package.
16+
17+
```
18+
$ git diff HEAD
19+
diff --git a/Package.swift b/Package.swift
20+
index 477bcbd..3e7bb06 100644
21+
--- a/Package.swift
22+
+++ b/Package.swift
23+
@@ -4,13 +4,14 @@ import PackageDescription
24+
let package = Package(
25+
name: "SparkConnectSwiftWebapp",
26+
platforms: [
27+
- .macOS(.v13)
28+
+ .macOS(.v15)
29+
],
30+
dependencies: [
31+
// 💧 A server-side Swift web framework.
32+
.package(url: "https://github.com/vapor/vapor.git", from: "4.110.1"),
33+
// 🔵 Non-blocking, event-driven networking for Swift. Used for custom executors
34+
.package(url: "https://github.com/apple/swift-nio.git", from: "2.65.0"),
35+
+ .package(url: "https://github.com/apache/spark-connect-swift.git", branch: "main"),
36+
],
37+
targets: [
38+
.executableTarget(
39+
@@ -19,6 +20,7 @@ let package = Package(
40+
.product(name: "Vapor", package: "vapor"),
41+
.product(name: "NIOCore", package: "swift-nio"),
42+
.product(name: "NIOPosix", package: "swift-nio"),
43+
+ .product(name: "SparkConnect", package: "spark-connect-swift"),
44+
],
45+
swiftSettings: swiftSettings
46+
),
47+
diff --git a/Sources/SparkConnectSwiftWebapp/routes.swift b/Sources/SparkConnectSwiftWebapp/routes.swift
48+
index 2edcc8f..22313c8 100644
49+
--- a/Sources/SparkConnectSwiftWebapp/routes.swift
50+
+++ b/Sources/SparkConnectSwiftWebapp/routes.swift
51+
@@ -1,4 +1,5 @@
52+
import Vapor
53+
+import SparkConnect
54+
55+
func routes(_ app: Application) throws {
56+
app.get { req async in
57+
@@ -6,6 +7,15 @@ func routes(_ app: Application) throws {
58+
}
59+
60+
app.get("hello") { req async -> String in
61+
- "Hello, world!"
62+
+ return await Task {
63+
+ do {
64+
+ let spark = try await SparkSession.builder.getOrCreate()
65+
+ let response = "Hi, this is powered by the Apache Spark \(await spark.version)."
66+
+ await spark.stop()
67+
+ return response
68+
+ } catch {
69+
+ return "Fail to connect: \(error)"
70+
+ }
71+
+ }.value
72+
}
73+
}
74+
```
75+
76+
## Run this Swift application.
77+
78+
```
79+
$ swift run
80+
```
81+
82+
## Connect to the Swift Web Server to talk with `Apache Spark`.
83+
84+
```
85+
$ curl http://127.0.0.1:8080/
86+
Welcome to the Swift world. Say hello!%
87+
$ curl http://127.0.0.1:8080/hello
88+
Hi, this is powered by the Apache Spark 4.0.0.%
89+
```

‎Examples/web/Sources/SparkConnectSwiftWeb/configure.swift‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
//
2+
// Licensed to the Apache Software Foundation (ASF) under one
3+
// or more contributor license agreements. See the NOTICE file
4+
// distributed with this work for additional information
5+
// regarding copyright ownership. The ASF licenses this file
6+
// to you under the Apache License, Version 2.0 (the
7+
// "License"); you may not use this file except in compliance
8+
// with the License. You may obtain a copy of the License at
9+
//
10+
// http://www.apache.org/licenses/LICENSE-2.0
11+
//
12+
// Unless required by applicable law or agreed to in writing,
13+
// software distributed under the License is distributed on an
14+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
// KIND, either express or implied. See the License for the
16+
// specific language governing permissions and limitations
17+
// under the License.
18+
//
19+
20+
import Vapor
21+
22+
// configures your application
23+
public func configure(_ app: Application) async throws {
24+
// register routes
25+
try routes(app)
26+
}

‎Examples/web/Sources/SparkConnectSwiftWeb/entrypoint.swift‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
//
2+
// Licensed to the Apache Software Foundation (ASF) under one
3+
// or more contributor license agreements. See the NOTICE file
4+
// distributed with this work for additional information
5+
// regarding copyright ownership. The ASF licenses this file
6+
// to you under the Apache License, Version 2.0 (the
7+
// "License"); you may not use this file except in compliance
8+
// with the License. You may obtain a copy of the License at
9+
//
10+
// http://www.apache.org/licenses/LICENSE-2.0
11+
//
12+
// Unless required by applicable law or agreed to in writing,
13+
// software distributed under the License is distributed on an
14+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
// KIND, either express or implied. See the License for the
16+
// specific language governing permissions and limitations
17+
// under the License.
18+
//
19+
20+
import Logging
21+
import NIOCore
22+
import NIOPosix
23+
import Vapor
24+
25+
@main
26+
enum Entrypoint {
27+
static func main() async throws {
28+
var env = try Environment.detect()
29+
try LoggingSystem.bootstrap(from: &env)
30+
31+
let app = try await Application.make(env)
32+
33+
// This attempts to install NIO as the Swift Concurrency global executor.
34+
// You can enable it if you'd like to reduce the amount of context switching between NIO and Swift Concurrency.
35+
// Note: this has caused issues with some libraries that use `.wait()` and cleanly shutting down.
36+
// If enabled, you should be careful about calling async functions before this point as it can cause assertion failures.
37+
// let executorTakeoverSuccess = NIOSingletons.unsafeTryInstallSingletonPosixEventLoopGroupAsConcurrencyGlobalExecutor()
38+
// app.logger.debug("Tried to install SwiftNIO's EventLoopGroup as Swift's global concurrency executor", metadata: ["success": .stringConvertible(executorTakeoverSuccess)])
39+
40+
do {
41+
try await configure(app)
42+
try await app.execute()
43+
} catch {
44+
app.logger.report(error: error)
45+
try? await app.asyncShutdown()
46+
throw error
47+
}
48+
try await app.asyncShutdown()
49+
}
50+
}

‎Examples/web/Sources/SparkConnectSwiftWeb/routes.swift‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
//
2+
// Licensed to the Apache Software Foundation (ASF) under one
3+
// or more contributor license agreements. See the NOTICE file
4+
// distributed with this work for additional information
5+
// regarding copyright ownership. The ASF licenses this file
6+
// to you under the Apache License, Version 2.0 (the
7+
// "License"); you may not use this file except in compliance
8+
// with the License. You may obtain a copy of the License at
9+
//
10+
// http://www.apache.org/licenses/LICENSE-2.0
11+
//
12+
// Unless required by applicable law or agreed to in writing,
13+
// software distributed under the License is distributed on an
14+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
// KIND, either express or implied. See the License for the
16+
// specific language governing permissions and limitations
17+
// under the License.
18+
//
19+
20+
import SparkConnect
21+
import Vapor
22+
23+
func routes(_ app: Application) throws {
24+
app.get { req async in
25+
"Welcome to the Swift world. Say hello!"
26+
}
27+
28+
app.get("hello") { req async -> String in
29+
return await Task {
30+
do {
31+
let spark = try await SparkSession.builder.getOrCreate()
32+
let response = "Hi, this is powered by the Apache Spark \(await spark.version)."
33+
await spark.stop()
34+
return response
35+
} catch {
36+
return "Fail to connect: \(error)"
37+
}
38+
}.value
39+
}
40+
}

‎README.md‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -108,10 +108,7 @@ SELECT * FROM t
108108
+----+
109109
```
110110
111-
You can find this command line example and additional `Swift Web Server` example in the following repositories.
112-
113-
- <https://github.com/dongjoon-hyun/spark-connect-swift-app>
114-
- <https://github.com/dongjoon-hyun/spark-connect-swift-web>
111+
You can find more complete examples including Web Server application in the `Examples` directory.
115112
116113
## How to use `Spark SQL REPL` via `Spark Connect for Swift`
117114

0 commit comments

Comments
 (0)