Skip to content

Commit bebd5e1

Browse files
committed
[SPARK-51572] Support binary type in show and collect
### What changes were proposed in this pull request? This PR aims to support `binary` type in `show` and `collect`. ### Why are the changes needed? Like the other client, binary type values shows in an array style. ```python $ bin/pyspark --remote sc://localhost:15002 >>> sql("SELECT binary('abc')").show() +----------+ | abc| +----------+ |[61 62 63]| +----------+ ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #26 from dongjoon-hyun/SPARK-51572. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent b70eeb7 commit bebd5e1

File tree

3 files changed

+8
-0
lines changed

3 files changed

+8
-0
lines changed

Sources/SparkConnect/DataFrame.swift

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,6 +173,9 @@ public actor DataFrame: Sendable {
173173
let str = column.array as! AsString
174174
if column.data.isNull(i) {
175175
values.append(nil)
176+
} else if column.data.type.info == ArrowType.ArrowBinary {
177+
let binary = str.asString(i).utf8.map { String(format: "%02x", $0) }.joined(separator: " ")
178+
values.append("[\(binary)]")
176179
} else {
177180
values.append(str.asString(i))
178181
}
@@ -201,6 +204,9 @@ public actor DataFrame: Sendable {
201204
let str = column.array as! AsString
202205
if column.data.isNull(i) {
203206
values.append("NULL")
207+
} else if column.data.type.info == ArrowType.ArrowBinary {
208+
let binary = str.asString(i).utf8.map { String(format: "%02x", $0) }.joined(separator: " ")
209+
values.append("[\(binary)]")
204210
} else {
205211
values.append(str.asString(i))
206212
}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
SELECT binary('abc')
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
[["[61 62 63]"]]

0 commit comments

Comments
 (0)