Skip to content

Commit 0bc78b4

Browse files
committed
[SPARK-51977] Improve SparkSQLRepl to support multiple lines
### What changes were proposed in this pull request? This PR aims to improve `SparkSQLRepl` example to support multiple lines. ### Why are the changes needed? To help users to check complex queries easily. As an example, SQL Pipe Syntax is added to the `README.md`. ```sql FROM ORC.`/opt/spark/examples/src/main/resources/users.orc` |> AGGREGATE COUNT(*) cnt GROUP BY name |> ORDER BY cnt DESC, name ASC ; ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? After passing the CIs, do manual check. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #102 from dongjoon-hyun/SPARK-51977. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent f913a2c commit 0bc78b4

File tree

2 files changed

+30
-2
lines changed

2 files changed

+30
-2
lines changed

README.md

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -182,6 +182,29 @@ spark-sql (default)> DROP DATABASE db1 CASCADE;
182182
spark-sql (default)> exit;
183183
```
184184
185+
Apache Spark 4 supports [SQL Pipe Syntax](https://dist.apache.org/repos/dist/dev/spark/v4.0.0-rc4-docs/_site/sql-pipe-syntax.html).
186+
187+
```
188+
$ swift run
189+
...
190+
Build of product 'SparkSQLRepl' complete! (2.33s)
191+
Connected to Apache Spark 4.0.0 Server
192+
spark-sql (default)>
193+
FROM ORC.`/opt/spark/examples/src/main/resources/users.orc`
194+
|> AGGREGATE COUNT(*) cnt
195+
GROUP BY name
196+
|> ORDER BY cnt DESC, name ASC
197+
;
198+
+------+---+
199+
| name|cnt|
200+
+------+---+
201+
|Alyssa| 1|
202+
| Ben| 1|
203+
+------+---+
204+
205+
Time taken: 159 ms
206+
```
207+
185208
You can use `SPARK_REMOTE` to specify the [Spark Connect connection string](https://spark.apache.org/docs/latest/spark-connect-overview.html#set-sparkremote-environment-variable) in order to provide more options.
186209
187210
```bash

Sources/SparkSQLRepl/main.swift

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,15 +26,20 @@ let spark = try await SparkSession.builder.getOrCreate()
2626
print("Connected to Apache Spark \(await spark.version) Server")
2727

2828
var isRunning = true
29+
var lines = ""
2930
while isRunning {
30-
print("spark-sql (\(try await spark.catalog.currentDatabase()))> ", terminator: "")
31+
if lines.isEmpty {
32+
print("spark-sql (\(try await spark.catalog.currentDatabase()))> ", terminator: "")
33+
}
3134
guard let input = readLine() else {
3235
isRunning = false
3336
break
3437
}
38+
lines += input + " "
3539

36-
let matches = input.matches(of: statement)
40+
let matches = lines.matches(of: statement)
3741
for match in matches {
42+
lines = ""
3843
switch match.1 {
3944
case "exit":
4045
isRunning = false

0 commit comments

Comments
 (0)