Commit 4185eb9
committed
### What changes were proposed in this pull request?
This PR aims to support `struct` type in `ArrowReader`.
### Why are the changes needed?
`ArrowReader` depends on `ArrowReaderHelper` which depends on `ProtoUtil.fromProto` function. This PR extends `fromProto` to handle `struct` type in order to avoid unknown type error in `ArrowReader` . The following is the result of `Scala` client.
```
$ bin/spark-connect-shell --remote sc://localhost:15002
...
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.0.0-preview2
/_/
Type in expressions to have them evaluated.
Spark session available as 'spark'.
scala> sql("SELECT struct(1), struct(2, struct(3))").show()
+---------+--------------------+
|struct(1)|struct(2, struct(3))|
+---------+--------------------+
| {1}| {2, {3}}|
+---------+--------------------+
```
### Does this PR introduce _any_ user-facing change?
No. There is no behavior change.
### How was this patch tested?
Pass the CIs with the newly added test cases.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #50 from dongjoon-hyun/SPARK-51763.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent ebffbbe commit 4185eb9
File tree
3 files changed
+4
-0
lines changed- Sources/SparkConnect
- Tests/SparkConnectTests/Resources/queries
3 files changed
+4
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
| 67 | + | |
| 68 | + | |
67 | 69 | | |
68 | 70 | | |
69 | 71 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
0 commit comments