Skip to content

MLE-24717 Bumping Spark and langchain and others #570

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 31, 2025
Merged

MLE-24717 Bumping Spark and langchain and others #570

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions gradle.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
version=3.0-SNAPSHOT
sparkVersion=4.1.0-preview2
sparkVersion=4.1.0-preview3
tikaVersion=3.2.3
semaphoreVersion=5.10.0
langchain4jVersion=1.5.0
langchain4jVersion=1.8.0

# Always have to make sure we use the same version of Jackson that Spark does.
jacksonVersion=2.20.0

# Define these on the command line to publish to OSSRH
# See https://central.sonatype.org/publish/publish-gradle/#credentials for more information
Expand Down
24 changes: 6 additions & 18 deletions marklogic-spark-connector/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,6 @@ configurations {
details.useVersion "3.1.12"
details.because "Bumping from 3.1.9 (what Spark SQL 4.0.1 depends on) to 3.1.12 to minimize CVEs."
}
if (details.requested.group.equals("io.netty") and details.requested.version.startsWith("4.1.1")) {
details.useVersion "4.1.128.Final"
details.because "Bumping from 4.1.127 (what Spark SQL 4.1.0-preview2 depends on) to minimize CVEs."
}
}

resolutionStrategy {
// Addresses CVE-2025-58457; Spark depends on 3.9.3, and the CVE is fixed in 3.9.4.
force "org.apache.zookeeper:zookeeper:3.9.4"
}
}
}
Expand All @@ -34,7 +25,7 @@ dependencies {
}

// This is compileOnly as Spark will provide its own copy at runtime.
compileOnly "com.fasterxml.jackson.core:jackson-databind:2.19.0"
compileOnly "com.fasterxml.jackson.core:jackson-databind:${jacksonVersion}"

implementation("com.marklogic:marklogic-client-api:8.0.0") {
// Need to use the versions of Jackson preferred by Spark.
Expand All @@ -61,14 +52,11 @@ dependencies {
implementation "org.apache.tika:tika-core:${tikaVersion}"

// Needed for using XmlMapper.
implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.19.0") {
// Not needed, as the modules in this group that this dependency depends on are all provided by Spark.
exclude group: "com.fasterxml.jackson.core"
}
implementation "com.fasterxml.jackson.dataformat:jackson-dataformat-xml:${jacksonVersion}"
Copy link

Copilot AI Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The exclude group statements for 'com.fasterxml.jackson.core' were removed, but they may still be needed to prevent version conflicts. Consider adding back the exclude statements to ensure Spark's Jackson versions take precedence.

Suggested change
implementation "com.fasterxml.jackson.dataformat:jackson-dataformat-xml:${jacksonVersion}"
implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-xml:${jacksonVersion}") {
exclude group: "com.fasterxml.jackson.core"
}

Copilot uses AI. Check for mistakes.

// Supports reading and writing RDF data. Including this here so it's available to the tests as well.
// Bumped to 5.x, which requires Java 17, while upgrading Spark to 4.x.
implementation("org.apache.jena:jena-arq:5.5.0") {
implementation("org.apache.jena:jena-arq:5.6.0") {
exclude group: "com.fasterxml.jackson.core"
exclude group: "com.fasterxml.jackson.dataformat"
}
Expand All @@ -82,7 +70,7 @@ dependencies {

// Need this so that an OkHttpClientConfigurator can be created.
// Only needs compileOnly, as the Java Client brings this as an implementation dependency.
compileOnly 'com.squareup.okhttp3:okhttp:5.2.0'
compileOnly 'com.squareup.okhttp3:okhttp:5.2.1'

// Automatic loading of test framework implementation dependencies is deprecated.
// https://docs.gradle.org/current/userguide/upgrading_version_8.html#test_framework_implementation_dependencies
Expand All @@ -95,7 +83,7 @@ dependencies {
}

// Supports testing the embedder feature.
testImplementation("dev.langchain4j:langchain4j-embeddings-all-minilm-l6-v2:1.5.0-beta11") {
testImplementation("dev.langchain4j:langchain4j-embeddings-all-minilm-l6-v2:1.8.0-beta15") {
exclude group: "com.fasterxml.jackson.core"
}

Expand All @@ -109,7 +97,7 @@ dependencies {
exclude group: "com.fasterxml.jackson.dataformat"
}

testImplementation "ch.qos.logback:logback-classic:1.5.19"
testImplementation "ch.qos.logback:logback-classic:1.5.20"
testImplementation "org.skyscreamer:jsonassert:1.5.3"

testImplementation "org.apache.tika:tika-parser-microsoft-module:${tikaVersion}"
Expand Down