Skip to content

Commit 4185eb9

Browse files
committed
[SPARK-51763] Support struct type in ArrowReader
### What changes were proposed in this pull request? This PR aims to support `struct` type in `ArrowReader`. ### Why are the changes needed? `ArrowReader` depends on `ArrowReaderHelper` which depends on `ProtoUtil.fromProto` function. This PR extends `fromProto` to handle `struct` type in order to avoid unknown type error in `ArrowReader` . The following is the result of `Scala` client. ``` $ bin/spark-connect-shell --remote sc://localhost:15002 ... Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-preview2 /_/ Type in expressions to have them evaluated. Spark session available as 'spark'. scala> sql("SELECT struct(1), struct(2, struct(3))").show() +---------+--------------------+ |struct(1)|struct(2, struct(3))| +---------+--------------------+ | {1}| {2, {3}}| +---------+--------------------+ ``` ### Does this PR introduce _any_ user-facing change? No. There is no behavior change. ### How was this patch tested? Pass the CIs with the newly added test cases. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #50 from dongjoon-hyun/SPARK-51763. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent ebffbbe commit 4185eb9

File tree

3 files changed

+4
-0
lines changed

3 files changed

+4
-0
lines changed

Sources/SparkConnect/ProtoUtil.swift

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,8 @@ func fromProto( // swiftlint:disable:this cyclomatic_complexity
6464
let arrowUnit: ArrowTime64Unit = timeType.unit == .microsecond ? .microseconds : .nanoseconds
6565
arrowType = ArrowTypeTime64(arrowUnit)
6666
}
67+
case .struct_:
68+
arrowType = ArrowType(ArrowType.ArrowStruct)
6769
default:
6870
arrowType = ArrowType(ArrowType.ArrowUnknown)
6971
}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
SELECT struct(1), struct(2, struct(3))
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
[["{1}","{2,{3}}"]]

0 commit comments

Comments
 (0)