Skip to content

Bump SLF4J from 1.7.30 to 2.0.16. #33574

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Abacn merged 18 commits into from
Feb 24, 2025
Merged

Bump SLF4J from 1.7.30 to 2.0.16. #33574

Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
4b124cb
Update slf4j version
shunping Jan 13, 2025
f3636ff
Remove slf4j from arrow dependency exclusion.
shunping Jan 13, 2025
440bd2a
Fix two test failures due to upgrade slf4j-jdk14 to 2.x
shunping Jan 14, 2025
0f4e5f1
Fixed another four failed tests.
shunping Jan 14, 2025
fd01768
Merge branch 'master' into upgrade-slf4j
shunping Feb 4, 2025
fd16148
Bump the default spark version from 3.2.2 to 3.5.0.
shunping Feb 4, 2025
bcc3465
Add used but not declared deps for spark 3.5.0
shunping Feb 4, 2025
43ddc02
Temporary modify spark version to 3.x in sparkreceiver.
shunping Feb 4, 2025
f452f05
Fix failed spark tests.
shunping Feb 5, 2025
40cdc9d
A better workaround for Spark 3.2.x
shunping Feb 5, 2025
b810034
Take out the add-opens for tests as they were only run in java 8 and 11.
shunping Feb 5, 2025
6ac9b97
Merge branch 'master' into upgrade-slf4j
shunping Feb 5, 2025
cc21ecd
Merge branch 'master' into upgrade-slf4j
shunping Feb 5, 2025
bdbb72e
Mention changes in CHANGES.md
shunping Feb 5, 2025
1710eff
Update comments
shunping Feb 5, 2025
d6b56c6
Move sparkReceiver/2 to sparkreceiver/3 that supports Spark 3.x.
shunping Feb 11, 2025
935a637
Minor fix on cdap spark dependency
shunping Feb 12, 2025
cb62acb
Merge branch 'master' into upgrade-slf4j
shunping Feb 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@
* Support for X source added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).

## New Features / Improvements

* [Java] Upgrade SLF4J to 2.0.16. Update default Spark version to 3.5.0. ([#33574](https://github.com/apache/beam/pull/33574))
* X feature added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).

## Breaking Changes
Expand Down
4 changes: 2 additions & 2 deletions buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -636,12 +636,12 @@ class BeamModulePlugin implements Plugin<Project> {
def quickcheck_version = "1.0"
def sbe_tool_version = "1.25.1"
def singlestore_jdbc_version = "1.1.4"
def slf4j_version = "1.7.30"
def slf4j_version = "2.0.16"
def snakeyaml_engine_version = "2.6"
def snakeyaml_version = "2.2"
def solace_version = "10.21.0"
def spark2_version = "2.4.8"
def spark3_version = "3.2.2"
def spark3_version = "3.5.0"
def spotbugs_version = "4.0.6"
def testcontainers_version = "1.19.7"
// [bomupgrader] determined by: org.apache.arrow:arrow-memory-core, consistent with: google_cloud_platform_libraries_bom
Expand Down
4 changes: 2 additions & 2 deletions runners/google-cloud-dataflow-java/worker/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ applyJavaNature(
// TODO(https://github.com/apache/beam/issues/19114): Move DataflowRunnerHarness class under org.apache.beam.runners.dataflow.worker namespace
"com/google/cloud/dataflow/worker/DataflowRunnerHarness.class",
// Allow slf4j implementation worker for logging during pipeline execution
"org/slf4j/impl/**"
"org/slf4j/jul/**"
],
generatedClassPatterns: [
/^org\.apache\.beam\.runners\.dataflow\.worker\.windmill.*/,
Expand Down Expand Up @@ -240,7 +240,7 @@ project.task('validateShadedJarContainsSlf4jJdk14', dependsOn: 'shadowJar') {
doLast {
project.configurations.shadow.artifacts.files.each {
FileTree slf4jImpl = project.zipTree(it).matching {
include "org/slf4j/impl/JDK14LoggerAdapter.class"
include "org/slf4j/jul/JDK14LoggerAdapter.class"
}
outFile.text = slf4jImpl.files
if (slf4jImpl.files.isEmpty()) {
Expand Down
20 changes: 20 additions & 0 deletions runners/spark/3/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,26 @@ sparkVersions.each { kv ->
}

dependencies {
// Spark versions prior to 3.4.0 are compiled against SLF4J 1.x. The
// `org.apache.spark.internal.Logging.isLog4j12()` function references an
// SLF4J 1.x binding class (org.slf4j.impl.StaticLoggerBinder) which is
// no longer available in SLF4J 2.x. This results in a
// `java.lang.NoClassDefFoundError`.
//
// The workaround is to provide an SLF4J 1.x binding module out of group
// `org.slf4j` to resolve the issue.
// Module `org.apache.logging.log4j:log4j-slf4j-impl` is an example that
// provides a compatible SLF4J 1.x binding regardless SLF4J upgrade.
// Binding/provider modules under group `org.slf4j` (e.g.,
// slf4j-simple, slf4j-reload4j) get upgraded as a new SLF4J version is in
// use, and therefore do not contain the 1.x binding classes.
//
// Notice that Spark 3.1.x uses `ch.qos.logback:logback-classic` and is
// unaffected by the SLF4J upgrade. Spark 3.3.x already uses
// `log4j-slf4j-impl` so it is also unaffected.
if ("$kv.key" >= "320" && "$kv.key" <= "324") {
"sparkVersion$kv.key" library.java.log4j2_slf4j_impl
}
spark.components.each { component -> "sparkVersion$kv.key" "$component:$kv.value" }
}

Expand Down
8 changes: 8 additions & 0 deletions runners/spark/spark_runner.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,10 @@ dependencies {
spark.components.each { component ->
provided "$component:$spark_version"
}
if ("$spark_version" >= "3.5.0") {
implementation "org.apache.spark:spark-common-utils_$spark_scala_version:$spark_version"
implementation "org.apache.spark:spark-sql-api_$spark_scala_version:$spark_version"
}
permitUnusedDeclared "org.apache.spark:spark-network-common_$spark_scala_version:$spark_version"
implementation "io.dropwizard.metrics:metrics-core:4.1.1" // version used by Spark 3.1
compileOnly "org.scala-lang:scala-library:2.12.15"
Expand All @@ -202,6 +206,10 @@ dependencies {
testImplementation library.java.mockito_core
testImplementation "org.assertj:assertj-core:3.11.1"
testImplementation "org.apache.zookeeper:zookeeper:3.4.11"
if ("$spark_version" >= "3.5.0") {
testImplementation "org.apache.spark:spark-common-utils_$spark_scala_version:$spark_version"
testImplementation "org.apache.spark:spark-sql-api_$spark_scala_version:$spark_version"
}
validatesRunner project(path: ":sdks:java:core", configuration: "shadowTest")
validatesRunner project(path: ":runners:core-java", configuration: "testRuntimeMigration")
validatesRunner project(":sdks:java:io:hadoop-format")
Expand Down
15 changes: 3 additions & 12 deletions sdks/java/extensions/arrow/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,19 +24,10 @@ description = "Apache Beam :: SDKs :: Java :: Extensions :: Arrow"
dependencies {
implementation library.java.vendored_guava_32_1_2_jre
implementation project(path: ":sdks:java:core", configuration: "shadow")
implementation(library.java.arrow_vector) {
// Arrow 15 has compile dependency of slf4j 2.x where Beam does not support
exclude group: 'org.slf4j', module: 'slf4j-api'
}
implementation(library.java.arrow_memory_core) {
// Arrow 15 has compile dependency of slf4j 2.x where Beam does not support
exclude group: 'org.slf4j', module: 'slf4j-api'
}
implementation(library.java.arrow_vector)
implementation(library.java.arrow_memory_core)
implementation library.java.joda_time
testImplementation(library.java.arrow_memory_netty) {
// Arrow 15 has compile dependency of slf4j 2.x where Beam does not support
exclude group: 'org.slf4j', module: 'slf4j-api'
}
testImplementation(library.java.arrow_memory_netty)
testImplementation library.java.junit
testImplementation library.java.hamcrest
testRuntimeOnly library.java.slf4j_simple
Expand Down
15 changes: 3 additions & 12 deletions sdks/java/io/google-cloud-platform/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,22 +138,13 @@ dependencies {
implementation library.java.slf4j_api
implementation library.java.vendored_grpc_1_69_0
implementation library.java.vendored_guava_32_1_2_jre
implementation(library.java.arrow_memory_core) {
// Arrow 15 has compile dependency of slf4j 2.x where Beam does not support
exclude group: 'org.slf4j', module: 'slf4j-api'
}
implementation(library.java.arrow_vector) {
// Arrow 15 has compile dependency of slf4j 2.x where Beam does not support
exclude group: 'org.slf4j', module: 'slf4j-api'
}
implementation library.java.arrow_memory_core
implementation library.java.arrow_vector

implementation 'com.google.http-client:google-http-client-gson:1.41.2'
implementation "org.threeten:threetenbp:1.4.4"

testImplementation(library.java.arrow_memory_netty) {
// Arrow 15 has compile dependency of slf4j 2.x where Beam does not support
exclude group: 'org.slf4j', module: 'slf4j-api'
}
testImplementation library.java.arrow_memory_netty
testImplementation project(path: ":sdks:java:core", configuration: "shadowTest")
testImplementation project(path: ":sdks:java:extensions:avro", configuration: "testRuntimeMigration")
testImplementation project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "testRuntimeMigration")
Expand Down
4 changes: 2 additions & 2 deletions sdks/java/io/sparkreceiver/2/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,8 @@ dependencies {
implementation library.java.commons_lang3
implementation library.java.joda_time
implementation library.java.slf4j_api
implementation library.java.spark_streaming
implementation library.java.spark_core
implementation library.java.spark3_streaming
Copy link
Copy Markdown
Contributor

@Abacn Abacn Feb 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's move sparkreceiver/2 to sparkreceiver/3 or simply sparkreceiver/. The package name is unchanged so that release is unchanged.

Copy link
Copy Markdown
Collaborator Author

@shunping shunping Feb 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I sent an email to dev list regarding this change. I will wait a couple days to see if there is any objection or other suggestion.

implementation library.java.spark3_core
implementation library.java.vendored_guava_32_1_2_jre
implementation project(path: ":sdks:java:core", configuration: "shadow")
compileOnly "org.scala-lang:scala-library:2.11.12"
Expand Down
Loading