[SPARK-52744] Add MacOS integration test with Apache Spark 4.1.0-preview1 RC1
#210
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dongjoon-hyun
force-pushed
the
SPARK-52744
branch
2 times, most recently
from
5d90e9a to
6e18f9e
Compare
dongjoon-hyun
force-pushed
the
SPARK-52744
branch
2 times, most recently
from
b3651c2 to
e8c02dc
Compare
|
Is it a real error? |
dongjoon-hyun
force-pushed
the
SPARK-52744
branch
from
e8c02dc to
8c287ce
Compare
|
No, @viirya. it seems to be a JVM time database different. |
dongjoon-hyun
commented
Jul 10, 2025
Sources/SparkConnect/DataFrame.swift
Outdated
| case .timeInfo(.timestamp): | ||
| let timestampType = column.data.type as! ArrowTypeTimestamp | ||
| assert(timestampType.timezone == "Etc/UTC") | ||
| assert(timestampType.timezone == "Etc/UTC" || timestampType.timezone == "GMT") |
There was a problem hiding this comment.
This is the updated part, @viirya .
dongjoon-hyun
force-pushed
the
SPARK-52744
branch
from
8c287ce to
290bf04
Compare
viirya
approved these changes
Jul 10, 2025
dongjoon-hyun
commented
Jul 10, 2025
| let spark = try await SparkSession.builder.getOrCreate() | ||
| let version = await spark.version | ||
| #expect(version.starts(with: "4.0.0") || version.starts(with: "3.5.")) | ||
| #expect(version.starts(with: "4.") || version.starts(with: "3.5.")) |
There was a problem hiding this comment.
This is updated. Previously, I didn't expect 4.1.x release in 2025.
|
Ur, sorry. It seems that the GitHub Action environment seems to have different from local. Let me dig a little more. Locally, |
dongjoon-hyun
force-pushed
the
SPARK-52744
branch
from
290bf04 to
5160124
Compare
dongjoon-hyun
commented
Jul 10, 2025
| @Test | ||
| func timestamp() async throws { | ||
| let spark = try await SparkSession.builder.getOrCreate() | ||
| // TODO(SPARK-52747) |
There was a problem hiding this comment.
Timestamp support is not released yet. I created SPARK-52747 to investigate timezone issue completely.
|
Thank you, @viirya . I'll merge this for now and do the follow-up for the TODO JIRAs. |
dongjoon-hyun
added a commit
that referenced
this pull request
Jul 10, 2025
### What changes were proposed in this pull request? This PR aims to support `defineDataset ` API in order to support `Declarative Pipelines` (SPARK-51727) of Apache Spark `4.1.0-preview1`. ### Why are the changes needed? To support the new feature incrementally. ### Does this PR introduce _any_ user-facing change? No, this is a new feature. ### How was this patch tested? Pass the CIs with `4.1.0-preview1` test pipeline because we added it. - #210 ### Was this patch authored or co-authored using generative AI tooling? No. Closes #211 from dongjoon-hyun/SPARK-52748. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
dongjoon-hyun
added a commit
that referenced
this pull request
Jul 10, 2025
### What changes were proposed in this pull request? This PR aims to support `defineFlow ` API in order to support `Declarative Pipelines` (SPARK-51727) of Apache Spark `4.1.0-preview1`. ### Why are the changes needed? To support the new feature incrementally. ### Does this PR introduce _any_ user-facing change? No, this is a new feature. ### How was this patch tested? Pass the CIs with `4.1.0-preview1` test pipeline. - #210 <img width="1000" height="373" alt="Screenshot 2025-07-10 at 07 25 37" src="https://github.com/user-attachments/assets/b4e214f6-de6c-4c31-8482-58e8de1dd4ff" /> ### Was this patch authored or co-authored using generative AI tooling? No. Closes #212 from dongjoon-hyun/SPARK-52756. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
This was referenced Jul 10, 2025
dongjoon-hyun
added a commit
that referenced
this pull request
Aug 25, 2025
…sting ### What changes were proposed in this pull request? This PR aims to use `release` build in Apache Spark `4.1.0-preview1` testing. ### Why are the changes needed? Although we used `release` build in CI since SPARK-52085, we missed to enable it when we add a new test job in SPARK-52744. - #136 - #210 To be consistent, we need to use release build in CIs. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #220 from dongjoon-hyun/SPARK-53374. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR aims to add
Apache Spark 4.1.0-preview1(RC1) test coverage.Why are the changes needed?
To be ready for
4.1.0-preview1release.Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass the CIs.
Was this patch authored or co-authored using generative AI tooling?
No.