Skip to content

Commit 8ba2806

Browse files
committed
[SPARK-52075] Add Examples.md
### What changes were proposed in this pull request? This PR aims to add `Examples.md` to `Documentation.docc` directory in order to provide an overview documentation of all examples. ### Why are the changes needed? Currently, `GettingStarted.md` is visible here. - https://swiftpackageindex.com/apache/spark-connect-swift/main/documentation/sparkconnect/gettingstarted `Examples.md` will be visible in the following link. - https://swiftpackageindex.com/apache/spark-connect-swift/main/documentation/sparkconnect/examples ### Does this PR introduce _any_ user-facing change? No, this is a doc-only change. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #133 from dongjoon-hyun/SPARK-52075. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent aee213f commit 8ba2806

File tree

1 file changed

+137
-0
lines changed

1 file changed

+137
-0
lines changed
Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
# Spark Connect Swift Examples
2+
3+
This document provides an overview of the example applications inside [Examples](https://github.com/apache/spark-connect-swift/tree/main/Examples) directory. These examples demonstrate how to use Spark Connect Swift to interact with Apache Spark in different contexts.
4+
5+
## Prepare Spark Connect Server
6+
7+
Start a Spark Connect Server:
8+
9+
```bash
10+
docker run -it --rm -p 15002:15002 apache/spark:4.0.0-preview2 bash -c "/opt/spark/sbin/start-connect-server.sh --wait -c spark.log.level=ERROR"
11+
```
12+
13+
## Basic Application Example
14+
15+
The basic application example demonstrates fundamental operations with Apache Spark Connect, including:
16+
- Connecting to a Spark server
17+
- Creating and manipulating tables with SQL
18+
- Using DataFrame operations
19+
- Reading and writing data in the ORC format
20+
21+
### Key Features
22+
- SQL execution for table operations
23+
- DataFrame transformations with filter operations
24+
- Data persistence with ORC format
25+
- Basic session management
26+
27+
### How to Run
28+
29+
Build and run the application:
30+
31+
```bash
32+
# Using Docker
33+
docker build -t apache/spark-connect-swift:app .
34+
docker run -it --rm -e SPARK_REMOTE=sc://host.docker.internal:15002 apache/spark-connect-swift:app
35+
36+
# From source code
37+
swift run
38+
```
39+
40+
## Pi Calculation Example
41+
42+
The Pi calculation example shows how to use Spark Connect Swift for computational tasks by calculating an approximation of π (pi) using the Monte Carlo method.
43+
44+
### Key Features
45+
- Command-line argument handling
46+
- Mathematical computations with Spark
47+
- Random number generation
48+
- Filtering and counting operations
49+
50+
### How to Run
51+
52+
Build and run the application:
53+
54+
```bash
55+
# Using Docker
56+
docker build -t apache/spark-connect-swift:pi .
57+
docker run --rm -e SPARK_REMOTE=sc://host.docker.internal:15002 apache/spark-connect-swift:pi
58+
59+
# From source code
60+
swift run
61+
```
62+
63+
## Streaming Example
64+
65+
The streaming example demonstrates how to process streaming data using Spark Connect Swift client, specifically for counting words from a network socket stream.
66+
67+
### Key Features
68+
- Stream processing with Spark Connect
69+
- Network socket data source
70+
- Word counting with string operations
71+
- Real-time console output
72+
73+
### How to Run
74+
75+
Start a Netcat server as the data source:
76+
77+
```bash
78+
nc -lk 9999
79+
```
80+
81+
Build and run the application:
82+
83+
```bash
84+
# Using Docker
85+
docker build -t apache/spark-connect-swift:stream .
86+
docker run --rm -e SPARK_REMOTE=sc://host.docker.internal:15002 -e TARGET_HOST=host.docker.internal apache/spark-connect-swift:stream
87+
88+
# From source code
89+
swift run
90+
```
91+
92+
Type text into the Netcat terminal to see real-time word counting from `Spark Connect Server` container.
93+
94+
## Web Application Example
95+
96+
The web application example showcases how to integrate Spark Connect Swift with a web server using the Vapor framework.
97+
98+
### Key Features
99+
- HTTP server integration with Vapor
100+
- REST API endpoints
101+
- Spark session management within web requests
102+
- Version information retrieval
103+
104+
### How to Run
105+
106+
Build and run the application:
107+
108+
```bash
109+
# Using Docker
110+
docker build -t apache/spark-connect-swift:web .
111+
docker run -it --rm -p 8080:8080 -e SPARK_REMOTE=sc://host.docker.internal:15002 apache/spark-connect-swift:web
112+
113+
# From source code
114+
swift run
115+
```
116+
117+
Access the web application:
118+
119+
```bash
120+
# Root endpoint
121+
$ curl http://127.0.0.1:8080/
122+
Welcome to the Swift world. Say hello!%
123+
124+
# Spark-powered endpoint
125+
curl http://127.0.0.1:8080/hello
126+
Hi, this is powered by the Apache Spark 4.0.0-preview2.%
127+
```
128+
129+
## Development Environment
130+
131+
All examples include:
132+
- A Dockerfile for containerized execution
133+
- A Package.swift file for Swift Package Manager configuration
134+
- A README.md with detailed instructions
135+
- Source code in the Sources directory
136+
137+
These examples are designed to be used with Apache Spark 4.0.0 or newer, using the Spark Connect protocol for client-server interaction.

0 commit comments

Comments
 (0)