Skip to content

Conversation

dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Apr 10, 2025

What changes were proposed in this pull request?

This PR aims to support struct type in ArrowReader.

Why are the changes needed?

ArrowReader depends on ArrowReaderHelper which depends on ProtoUtil.fromProto function. This PR extends fromProto to handle struct type in order to avoid unknown type error in ArrowReader . The following is the result of Scala client.

$ bin/spark-connect-shell --remote sc://localhost:15002
...
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 4.0.0-preview2
      /_/

Type in expressions to have them evaluated.
Spark session available as 'spark'.

scala> sql("SELECT struct(1), struct(2, struct(3))").show()
+---------+--------------------+
|struct(1)|struct(2, struct(3))|
+---------+--------------------+
|      {1}|            {2, {3}}|
+---------+--------------------+

Does this PR introduce any user-facing change?

No. There is no behavior change.

How was this patch tested?

Pass the CIs with the newly added test cases.

Was this patch authored or co-authored using generative AI tooling?

No.

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-51763] Support ArrowStruct type [SPARK-51763] Support struct type in ArrowReader Apr 10, 2025
@dongjoon-hyun
Copy link
Member Author

Thank you, @viirya . Merged to main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants