[SPARK-54597][BUILD][FOLLOW-UP] Explicitly exclude org.lz4:lz4-java in SBT build

HyukjinKwon · dongjoon-hyun · commit 703da9ded7fc · 2025-12-22T14:47:39.000+09:00
### What changes were proposed in this pull request? This PR is a followup of #53327 that explicitly exclude lz4-java in SBT build. ### Why are the changes needed? For some reasons, SBT still tries to look for it: ``` 2025-12-21T08:16:32.3447761Z [info] Jar hash: 61bb3bb74c3d32b7ae527652d9d8c46efa6d04fc 2025-12-21T08:16:33.2910680Z [error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts: 2025-12-21T08:16:33.2912312Z [error] file:/home/spark-rm/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar: not found: /home/spark-rm/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar 2025-12-21T08:16:33.2913430Z [error] 2025-12-21T08:16:33.2914325Z [error] at lmcoursier.internal.shaded.coursier.Artifacts$.$anonfun$fetchArtifacts$9(Artifacts.scala:365) 2025-12-21T08:16:33.2915570Z [error] at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$extension$1(Task.scala:14) 2025-12-21T08:16:33.2916784Z [error] at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$extension$1$adapted(Task.scala:14) 2025-12-21T08:16:33.2917884Z [error] at lmcoursier.internal.shaded.coursier.util.Task$.wrap(Task.scala:82) 2025-12-21T08:16:33.2918859Z [error] at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$2(Task.scala:14) 2025-12-21T08:16:33.2919771Z [error] at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307) 2025-12-21T08:16:33.2920635Z [error] at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:51) 2025-12-21T08:16:33.2921512Z [error] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74) 2025-12-21T08:16:33.2922869Z [error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) 2025-12-21T08:16:33.2924071Z [error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) 2025-12-21T08:16:33.2925145Z [error] at java.base/java.lang.Thread.run(Thread.java:840) 2025-12-21T08:16:33.2926563Z [error] Caused by: lmcoursier.internal.shaded.coursier.cache.ArtifactError$NotFound: not found: /home/spark-rm/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar 2025-12-21T08:16:33.2928288Z [error] at lmcoursier.internal.shaded.coursier.cache.internal.Downloader.$anonfun$checkFileExists$1(Downloader.scala:603) 2025-12-21T08:16:33.2929450Z [error] at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) 2025-12-21T08:16:33.2930146Z [error] at scala.util.Success.$anonfun$map$1(Try.scala:255) 2025-12-21T08:16:33.2930723Z [error] at scala.util.Success.map(Try.scala:213) 2025-12-21T08:16:33.2931387Z [error] at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) 2025-12-21T08:16:33.2932190Z [error] at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:42) 2025-12-21T08:16:33.2933052Z [error] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74) 2025-12-21T08:16:33.2934069Z [error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) 2025-12-21T08:16:33.2938645Z [error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) 2025-12-21T08:16:33.2939423Z [error] at java.base/java.lang.Thread.run(Thread.java:840) 2025-12-21T08:16:33.2940265Z [error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts: 2025-12-21T08:16:33.2941556Z [error] file:/home/spark-rm/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar: not found: /home/spark-rm/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar 2025-12-21T08:16:33.2942421Z [error] 2025-12-21T08:16:33.2943007Z [error] at lmcoursier.internal.shaded.coursier.Artifacts$.$anonfun$fetchArtifacts$9(Artifacts.scala:365) 2025-12-21T08:16:33.2944078Z [error] at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$extension$1(Task.scala:14) 2025-12-21T08:16:33.2945450Z [error] at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$extension$1$adapted(Task.scala:14) 2025-12-21T08:16:33.2946441Z [error] at lmcoursier.internal.shaded.coursier.util.Task$.wrap(Task.scala:82) 2025-12-21T08:16:33.2947312Z [error] at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$2(Task.scala:14) 2025-12-21T08:16:33.2948105Z [error] at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307) 2025-12-21T08:16:33.2948811Z [error] at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:51) 2025-12-21T08:16:33.2949547Z [error] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74) 2025-12-21T08:16:33.2950403Z [error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) 2025-12-21T08:16:33.2951391Z [error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) 2025-12-21T08:16:33.2952135Z [error] at java.base/java.lang.Thread.run(Thread.java:840) 2025-12-21T08:16:33.2953218Z [error] Caused by: lmcoursier.internal.shaded.coursier.cache.ArtifactError$NotFound: not found: /home/spark-rm/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar 2025-12-21T08:16:33.2954841Z [error] at lmcoursier.internal.shaded.coursier.cache.internal.Downloader.$anonfun$checkFileExists$1(Downloader.scala:603) 2025-12-21T08:16:33.2955801Z [error] at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) 2025-12-21T08:16:33.2956376Z [error] at scala.util.Success.$anonfun$map$1(Try.scala:255) 2025-12-21T08:16:33.2956861Z [error] at scala.util.Success.map(Try.scala:213) 2025-12-21T08:16:33.2957389Z [error] at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) 2025-12-21T08:16:33.2958305Z [error] at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:42) 2025-12-21T08:16:33.2959058Z [error] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74) 2025-12-21T08:16:33.2959915Z [error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) 2025-12-21T08:16:33.2960919Z [error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) 2025-12-21T08:16:33.2961677Z [error] at java.base/java.lang.Thread.run(Thread.java:840) 2025-12-21T08:16:33.2996977Z [error] (streaming-kafka-0-10 / update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts: 2025-12-21T08:16:33.2998744Z [error] file:/home/spark-rm/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar: not found: /home/spark-rm/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar 2025-12-21T08:16:33.3000432Z [error] (sql-kafka-0-10 / update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts: 2025-12-21T08:16:33.3002097Z [error] file:/home/spark-rm/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar: not found: /home/spark-rm/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar 2025-12-21T08:16:33.3032908Z [error] Total time: 361 s (0:06:01.0), completed Dec 21, 2025, 8:16:33 AM ``` which seems breaking the release build https://github.com/apache/spark/actions/workflows/release.yml ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I cannot reproduce properly in my local. This is the fix assuming from the log. I will monitor the build. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #53556 from HyukjinKwon/SPARK-54597-followup. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
@@ -1214,6 +1214,7 @@ object ExcludedDependencies {
     libraryDependencies ~= { libs => libs.filterNot(_.name == "groovy-all") },
     excludeDependencies ++= Seq(
       ExclusionRule(organization = "ch.qos.logback"),
+      ExclusionRule("org.lz4", "lz4-java"),
       ExclusionRule("org.slf4j", "slf4j-simple"),
       ExclusionRule("javax.servlet", "javax.servlet-api"))
   )

Original file line number	Diff line number	Diff line change
`@@ -1214,6 +1214,7 @@ object ExcludedDependencies {`
`1214`	`1214`	`libraryDependencies ~= { libs => libs.filterNot(_.name == "groovy-all") },`
`1215`	`1215`	`excludeDependencies ++= Seq(`
`1216`	`1216`	`ExclusionRule(organization = "ch.qos.logback"),`
	`1217`	`+ ExclusionRule("org.lz4", "lz4-java"),`
`1217`	`1218`	`ExclusionRule("org.slf4j", "slf4j-simple"),`
`1218`	`1219`	`ExclusionRule("javax.servlet", "javax.servlet-api"))`
`1219`	`1220`	`)`