Skip to content

Conversation

dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Aug 26, 2025

What changes were proposed in this pull request?

This PR aims to recover MacOS-based integration tests by adding SPARK_LOCAL_IP environment variables like Apache Spark main repository.

Why are the changes needed?

To recover MacOS-based CI failures. Currently, all MacOS-based integration test GitHub Action jobs are broken.

The root cause is that SparkContext fails due to Invalid Spark URL error like the following because getHost is null in Java.

$ jshell
jshell> new java.net.URI("spark://HeartbeatReceiver@sat12-bq147_186fa0a6-15fa-47f9-ad18-71676ed8dcea-4A992716D7F1.local:49174").getHost()
$1 ==> null
25/08/26 06:20:49 INFO SparkConnectServer: Starting Spark session.
25/08/26 06:20:50 INFO SparkContext: Running Spark version 4.1.0-preview1
25/08/26 06:20:50 INFO SparkContext: OS info Mac OS X, 15.5, aarch64
25/08/26 06:20:50 INFO SparkContext: Java version 21.0.8
...
25/08/26 06:20:50 INFO JettyUtils: Start Jetty 0.0.0.0:4040 for SparkUI
25/08/26 06:20:50 INFO Utils: Successfully started service 'SparkUI' on port 4040.
...
25/08/26 06:20:50 INFO Executor: Starting executor ID driver on host sat12-bq147_186fa0a6-15fa-47f9-ad18-71676ed8dcea-4A992716D7F1.local
...
25/08/26 06:20:50 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Invalid Spark URL: spark://HeartbeatReceiver@sat12-bq147_186fa0a6-15fa-47f9-ad18-71676ed8dcea-4A992716D7F1.local:49174
...
25/08/26 06:20:50 INFO SparkContext: SparkContext is stopping with exitCode 0 from main at NativeMethodAccessorImpl.java:0.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs.

Was this patch authored or co-authored using generative AI tooling?

No.

@dongjoon-hyun
Copy link
Member Author

Could you review this PR, @viirya ? It seems that I found the root cause and fix.

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-53380] Add SPARK_LOCAL_IP env variable for integration tests on MacOS [SPARK-53380] Add SPARK_LOCAL_IP env variable to MacOS-based integration test jobs Aug 26, 2025
@dongjoon-hyun
Copy link
Member Author

Let me merge this to recover the CI~

@dongjoon-hyun dongjoon-hyun deleted the SPARK_LOCAL_IP branch August 26, 2025 06:51
Copy link
Member

@viirya viirya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

@dongjoon-hyun
Copy link
Member Author

Thank you, @viirya .

@peter-toth
Copy link

Late LGTM.

@dongjoon-hyun
Copy link
Member Author

Thank you, @peter-toth .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants