[SPARK-52742] Support createDataflowGraph

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to support createDataflowGraph API in order to support Declarative Pipelines (SPARK-51727) of Apache Spark 4.1.0-preview1.

Why are the changes needed?

To support the new feature incrementally.

Does this PR introduce any user-facing change?

No, this is a new feature.

How was this patch tested?

Manually run Apache Spark 4.1.0-preview1 RC1 .

$ sbin/start-connect-server.sh

Run the newly added unit test.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

cc @viirya and @peter-toth

[dongjoon-hyun]

Could you review this PR, @HyukjinKwon ? I'm catching up the new features of 4.1.0-preview1.

[HyukjinKwon]

cc @sryza

[dongjoon-hyun]

Thank you, @HyukjinKwon . Merged to main.

[viirya]

Looks good to me.

[dongjoon-hyun]

Thank you, @viirya .

[sryza]

@dongjoon-hyun – not directly related to this PR, but wanted to say:

1. Awesome to see support for declarative pipelines starting to get added to Spark Connect Swift
2. If you're planning to also add support for DefineFlow, I wanted to give you a heads up that I'm aiming to post an additional change to it before it gets locked in for 4.1. It will likely involve nesting the relation field inside some sort of QueryFunctionResult message. https://issues.apache.org/jira/browse/SPARK-52762 explains the rationale.

[dongjoon-hyun]

No problem at all, @sryza . I fully understand preview semantic. Nothing is final here.

This aims to be technologically ready for the final delivery. :)

[dongjoon-hyun]

Just FYI, the basic framework was already merged with the first implementation like the following.

• [SPARK-52742] Support createDataflowGraph #208
• [SPARK-52743] Support startRun #209
• [SPARK-52744] Add MacOS integration test with Apache Spark 4.1.0-preview1 RC1 #210
• [SPARK-52748] Support defineDataset #211
• [SPARK-52756] Support defineFlow #212
• [SPARK-52758] Support defineSqlGraphElements #213

This Spark Connect for Swift development is supposed to give an early feedback to the Apache Spark main repository in the time frame of Apache Spark 4.1.0, @sryza .