Merge pull request #495 from marklogic/release/2.6.0

Merge release/2.6.0 into master

Author: rjrudin

.gitignore

@@ -20,3 +20,4 @@ docker/marklogic
 docker/sonarqube/data
 docker/sonarqube/logs
 export
+.run

CODEOWNERS

@@ -2,4 +2,4 @@
 # Each line is a file pattern followed by one or more owners.
 
 # These owners will be the default owners for everything in the repo.
-* @anu3990 @billfarber @rjrudin
+* @anu3990 @billfarber @rjrudin @stevebio

CONTRIBUTING.md

@@ -1,8 +1,12 @@
 This guide covers how to develop and test this project. It assumes that you have cloned this repository to your local
 workstation.
 
-You must use Java 11 or higher for developing, testing, and building this project. If you wish to use Sonar as 
-described below, you must use Java 17 or higher.
+**You must use Java 17 for developing, testing, and building this project**, even though the connector supports
+running on Java 11. For users, Java 17 is only required if using the splitting and embedding features, as those
+depend on a third party module that requires Java 17.
+
+**You also need Java 11 installed** so that the subprojects in this repository that require Java 11 have access to a 
+Java 11 SDK. [sdkman](https://sdkman.io/) is highly recommend for installing multiple JDKs.
 
 # Setup
 
@@ -40,49 +44,37 @@ To run the tests against the test application, run the following Gradle task:
 
     ./gradlew test
 
-## Generating code quality reports with SonarQube
-
-In order to use SonarQube, you must have used Docker to run this project's `docker-compose.yml` file, and you must
-have the services in that file running and you must use Java 17 to run the Gradle `sonar` task.
-
-To configure the SonarQube service, perform the following steps:
+**To run the tests in Intellij**, you must configure your JUnit template to include a few JVM args:
 
-1. Go to http://localhost:9000 .
-2. Login as admin/admin. SonarQube will ask you to change this password; you can choose whatever you want ("password" works).
-3. Click on "Create project manually".
-4. Enter "marklogic-spark" for the Project Name; use that as the Project Key too.
-5. Enter "develop" as the main branch name.
-6. Click on "Next".
-7. Click on "Use the global setting" and then "Create project".
-8. On the "Analysis Method" page, click on "Locally".
-9. In the "Provide a token" panel, click on "Generate". Copy the token.
-10. Add `systemProp.sonar.login=your token pasted here` to `gradle-local.properties` in the root of your project, creating
-that file if it does not exist yet.
+1. Go to Run -> Edit Configurations.
+2. Delete any JUnit configurations you already have.
+3. Click on "Edit configuration templates" and click on "JUnit".
+4. Click on "Modify options" and select "Add VM options" if it's not already selected. 
+5. In the VM options text input, add the following:
+   --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-exports=java.base/sun.security.action=ALL-UNNAMED
+6. Click "Apply".
+7. In the dropdown that has "Class" selected, change that to "Method" and hit "Apply" again.
 
-To run SonarQube, run the following Gradle tasks using Java 17, which will run all the tests with code coverage and 
-then generate a quality report with SonarQube:
+You may need to repeat steps 6 and 7. I've found Intellij to be a little finicky with actually applying these changes.
 
-    ./gradlew test sonar
+The net effect should be that when you run a JUnit class or method or suite of tests, those VM options are automatically
+added to the run configuration that Intellij creates for the class/method/suite. Those VM options are required to give
+Spark access to certain JVM modules. They are applied automatically when running the tests via Gradle.
 
-If you do not add `systemProp.sonar.login` to your `gradle-local.properties` file, you can specify the token via the
-following:
+**Alternatively**, you can open Preferences in Intellij and go to 
+"Build, Execution, and Deployment" -> "Build Tools" -> "Gradle". Then change "Build and run using" and "Run tests using"
+to "Gradle". This should result in Intellij using the `test` configuration in the `marklogic-spark-connector/build.gradle`
+file that registers the required JVM options, allowing for tests to run on Java 17.
 
-    ./gradlew test sonar -Dsonar.login=paste your token here
+## Testing text classification
 
-When that completes, you will see a line like this near the end of the logging:
+See the `ClassifyAdHocTest` class for instructions on how to test the text classification feature with a 
+valid connection to Semaphore.
 
-    ANALYSIS SUCCESSFUL, you can find the results at: http://localhost:9000/dashboard?id=marklogic-spark
-
-Click on that link. If it's the first time you've run the report, you'll see all issues. If you've run the report
-before, then SonarQube will show "New Code" by default. That's handy, as you can use that to quickly see any issues
-you've introduced on the feature branch you're working on. You can then click on "Overall Code" to see all issues.
-
-Note that if you only need results on code smells and vulnerabilities, you can repeatedly run `./gradlew sonar`
-without having to re-run the tests.
-
-You can also force Gradle to run `sonar` if any tests fail:
+## Generating code quality reports with SonarQube
 
-    ./gradlew clean test sonar --continue
+Please see our internal Wiki page - search for "Developer Experience SonarQube" - 
+for information on setting up SonarQube and using it with this repository.
 
 # Testing with PySpark
 
@@ -98,7 +90,7 @@ This will produce a single jar file for the connector in the `./build/libs` dire
 
 You can then launch PySpark with the connector available via:
 
-    pyspark --jars marklogic-spark-connector/build/libs/marklogic-spark-connector-2.5-SNAPSHOT.jar
+    pyspark --jars marklogic-spark-connector/build/libs/marklogic-spark-connector-2.6-SNAPSHOT.jar
 
 The below command is an example of loading data from the test application deployed via the instructions at the top of 
 this page. 
@@ -158,8 +150,8 @@ spark.read.option("header", True).csv("marklogic-spark-connector/src/test/resour
 When you run PySpark, it will create its own Spark cluster. If you'd like to try against a separate Spark cluster
 that still runs on your local machine, perform the following steps:
 
-1. Use [sdkman to install Spark](https://sdkman.io/sdks#spark). Run `sdk install spark 3.4.3` since we are currently
-building against Spark 3.4.3.
+1. Use [sdkman to install Spark](https://sdkman.io/sdks#spark). Run `sdk install spark 3.5.5` since we are currently
+building against Spark 3.5.5.
 2. `cd ~/.sdkman/candidates/spark/current/sbin`, which is where sdkman will install Spark.
 3. Run `./start-master.sh` to start a master Spark node.
 4. `cd ../logs` and open the master log file that was created to find the address for the master node. It will be in a
@@ -174,7 +166,7 @@ The Spark master GUI is at <http://localhost:8080>. You can use this to view det
 
 Now that you have a Spark cluster running, you just need to tell PySpark to connect to it:
 
-    pyspark --master spark://NYWHYC3G0W:7077 --jars marklogic-spark-connector/build/libs/marklogic-spark-connector-2.5-SNAPSHOT.jar
+    pyspark --master spark://NYWHYC3G0W:7077 --jars marklogic-spark-connector/build/libs/marklogic-spark-connector-2.6-SNAPSHOT.jar
 
 You can then run the same commands as shown in the PySpark section above. The Spark master GUI will allow you to 
 examine details of each of the commands that you run.
@@ -193,15 +185,15 @@ You will need the connector jar available, so run `./gradlew clean shadowJar` if
 You can then run a test Python program in this repository via the following (again, change the master address as 
 needed); note that you run this outside of PySpark, and `spark-submit` is available after having installed PySpark:
 
-    spark-submit --master spark://NYWHYC3G0W:7077 --jars marklogic-spark-connector/build/libs/marklogic-spark-connector-2.5-SNAPSHOT.jar marklogic-spark-connector/src/test/python/test_program.py
+    spark-submit --master spark://NYWHYC3G0W:7077 --jars marklogic-spark-connector/build/libs/marklogic-spark-connector-2.6-SNAPSHOT.jar marklogic-spark-connector/src/test/python/test_program.py
 
 You can also test a Java program. To do so, first move the `com.marklogic.spark.TestProgram` class from `src/test/java`
 to `src/main/java`. Then run the following:
 
 ```
 ./gradlew clean shadowJar
 cd marklogic-spark-connector
-spark-submit --master spark://NYWHYC3G0W:7077 --class com.marklogic.spark.TestProgram build/libs/marklogic-spark-connector-2.5-SNAPSHOT.jar
+spark-submit --master spark://NYWHYC3G0W:7077 --class com.marklogic.spark.TestProgram build/libs/marklogic-spark-connector-2.6-SNAPSHOT.jar
 ```
 
 Be sure to move `TestProgram` back to `src/test/java` when you are done. 

Jenkinsfile

@@ -8,6 +8,7 @@ def runtests(String javaVersion){
     cd marklogic-spark-connector
     echo "Waiting for MarkLogic server to initialize."
     sleep 30s
+    ./gradlew clean
    ./gradlew -i mlDeploy
    echo "Loading data a second time to try to avoid Optic bug with duplicate rows being returned."
    ./gradlew -i mlLoadData
@@ -85,8 +86,9 @@ pipeline{
           export JAVA_HOME=$JAVA17_HOME_DIR
           export GRADLE_USER_HOME=$WORKSPACE/$GRADLE_DIR
           export PATH=$GRADLE_USER_HOME:$JAVA_HOME/bin:$PATH
-          cp ~/.gradle/gradle.properties $GRADLE_USER_HOME;
           cd marklogic-spark-connector
+          ./gradlew clean
+          cp ~/.gradle/gradle.properties $GRADLE_USER_HOME;
            ./gradlew publish
         '''
       }

NOTICE.txt

@@ -13,17 +13,20 @@ Third Party Notices
 jackson-dataformat-xml 2.15.2 (Apache-2.0)
 jdom2 2.0.6.1 (Apache-2.0)
 jena-arq 4.10.0 (Apache-2.0)
-langchain4j 0.35.0 (Apache-2.0)
+langchain4j 1.0.0-beta2 (Apache-2.0)
 marklogic-client-api 7.1.0 (Apache-2.0)
 okhttp 4.12.0 (Apache-2.0)
+Semaphore-CS-Client 5.6.1 (Apache-2.0)
+Semaphore-Cloud-Client 5.6.1 (Apache-2.0)
+tika-core 3.1.0 (Apache-2.0)
 
 Common Licenses
 
 Apache License 2.0 (Apache-2.0)
 
 Third-Party Components
 
-The following is a list of the third-party components used by the MarkLogic® Spark connector 2.5.1 (last updated January 7, 2025):
+The following is a list of the third-party components used by the MarkLogic® Spark connector 2.6.0 (last updated May 1, 2025):
 
 jackson-dataformat-xml 2.15.2 (Apache-2.0)
 https://repo1.maven.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-xml/
@@ -37,7 +40,7 @@ jena-arq 4.10.0 (Apache-2.0)
 https://repo1.maven.org/maven2/org/apache/jena/jena-arq/
 For the full text of the Apache-2.0 license, see Apache License 2.0 (Apache-2.0)
 
-langchain4j 0.35.0 (Apache-2.0)
+langchain4j 1.0.0-beta2 (Apache-2.0)
 https://repo1.maven.org/maven2/dev/langchain4j/langchain4j/
 For the full text of the Apache-2.0 license, see Apache License 2.0 (Apache-2.0)
 
@@ -49,10 +52,21 @@ okhttp 4.12.0 (Apache-2.0)
 https://repo1.maven.org/maven2/com/squareup/okhttp3/okhttp/
 For the full text of the Apache-2.0 license, see Apache License 2.0 (Apache-2.0)
 
+Semaphore-CS-Client 5.6.1 (Apache-2.0)
+https://repo1.maven.org/maven2/com/smartlogic/csclient/Semaphore-CS-Client/
+For the full text of the Apache-2.0 license, see Apache License 2.0 (Apache-2.0)
+
+Semaphore-CS-Client 5.6.1 (Apache-2.0)
+https://repo1.maven.org/maven2/com/smartlogic/cloud/Semaphore-Cloud-Client/
+For the full text of the Apache-2.0 license, see Apache License 2.0 (Apache-2.0)
+
+tika-core 3.1.0 (Apache-2.0)
+https://repo1.maven.org/maven2/org/apache/tika/tika-core/
+For the full text of the Apache-2.0 license, see Apache License 2.0 (Apache-2.0)
 
 Common Licenses
 
-The following is a list of the third-party components used by the MarkLogic® Spark connector 2.5.1 (last updated January 7, 2025):
+The following is a list of the third-party components used by the MarkLogic® Spark connector 2.6.0 (last updated May 1, 2025):
 
 Apache License 2.0 (Apache-2.0)
 https://spdx.org/licenses/Apache-2.0.html

build.gradle

@@ -1,4 +1,5 @@
 plugins {
+  id 'net.saliman.properties' version '1.5.2'
   id "java-library"
   id "org.sonarqube" version "6.0.1.5171"
 }
@@ -12,15 +13,25 @@ sonar {
 }
 
 subprojects {
+  apply plugin: "net.saliman.properties"
   apply plugin: "java-library"
   apply plugin: "jacoco"
 
   group = "com.marklogic"
-  version "2.5.1"
+  version "2.6.0"
 
+  // Defaults to the Java 11 toolchain. Overridden by subprojects that require Java 17.
+  // See https://docs.gradle.org/current/userguide/toolchains.html .
   java {
-    sourceCompatibility = 11
-    targetCompatibility = 11
+    toolchain {
+      languageVersion = JavaLanguageVersion.of(11)
+    }
+  }
+
+  // Allows for quickly identifying compiler warnings.
+  tasks.withType(JavaCompile) {
+    options.compilerArgs << '-Xlint:unchecked'
+    options.deprecation = true
   }
 
   repositories {
@@ -32,11 +43,36 @@ subprojects {
   }
 
   configurations.all {
-    // Ensures that slf4j-api 1.x does not appear on the Flux classpath in particular, which can lead to this
-    // issue - https://www.slf4j.org/codes.html#StaticLoggerBinder .
+    resolutionStrategy.eachDependency { DependencyResolveDetails details ->
+      // Added after upgrading langchain4j to 1.0.0-beta2, which brought Jackson 2.18.2 in.
+      if (details.requested.group.startsWith('com.fasterxml.jackson')) {
+        details.useVersion '2.15.2'
+        details.because 'Need to match the version used by Spark.'
+      }
+      if (details.requested.group.equals("org.slf4j")) {
+        details.useVersion "2.0.16"
+        details.because "Ensures that slf4j-api 1.x does not appear on the Flux classpath in particular, which can " +
+          "lead to this issue - https://www.slf4j.org/codes.html#StaticLoggerBinder."
+      }
+      if (details.requested.group.equals("org.apache.logging.log4j")) {
+        details.useVersion "2.24.3"
+        details.because "Need to match the version used by Apache Tika. Spark uses 2.20.0 but automated tests confirm " +
+          "that Spark seems fine with 2.24.3."
+      }
+    }
+
     resolutionStrategy {
-      force "org.slf4j:slf4j-api:2.0.13"
+      // Avoids a classpath conflict between Spark and the tika-parser-microsoft-module. Tika needs a
+      // more recent version and Spark (and Jena as well) both seems fine with this (as they should be per semver).
+      force "org.apache.commons:commons-compress:1.27.1"
     }
+
+    // Excluded from Flux for size reasons, so excluded here as well to ensure we don't need it when running tests.
+    exclude module: "rocksdbjni"
+  }
+
+  task allDeps(type: DependencyReportTask) {
+    description = "Allows for generating dependency reports for every subproject in a single task."
   }
 
   test {
@@ -46,6 +82,9 @@ subprojects {
       events 'started', 'passed', 'skipped', 'failed'
       exceptionFormat 'full'
     }
+    environment "SEMAPHORE_API_KEY", semaphoreApiKey
+    environment "SEMAPHORE_HOST", semaphoreHost
+    environment "SEMAPHORE_PATH", semaphorePath
   }
 
   // See https://docs.gradle.org/current/userguide/jacoco_plugin.html .
@@ -55,4 +94,17 @@ subprojects {
       xml.required = true
     }
   }
+
+  // Turning off coverage verification, as we haven't figured out how to configure Sonar correctly to pick up the jacoco
+  // report from the "code-coverage-report" subproject and accurately capture coverage.
+  // See https://docs.gradle.org/current/userguide/jacoco_plugin.html#sec:jacoco_report_violation_rules
+  jacocoTestCoverageVerification {
+    violationRules {
+      rule {
+        limit {
+          minimum = 0.0
+        }
+      }
+    }
+  }
 }

code-coverage-report/build.gradle

@@ -6,10 +6,13 @@ plugins {
 }
 
 dependencies {
-  jacocoAggregation project(':marklogic-langchain4j')
   jacocoAggregation project(':marklogic-spark-api')
-  jacocoAggregation project(':marklogic-spark-langchain4j')
+  jacocoAggregation project(':marklogic-langchain4j')
   jacocoAggregation project(':marklogic-spark-connector')
+
+  // With the separation of tests into this project, we aren't getting any jacoco code coverage data.
+  // A problem to be fixed later.
+  jacocoAggregation project(':tests')
 }
 
 reporting {

docker-compose.yaml

@@ -1,4 +1,4 @@
-name: marklogic_spark
+name: docker-tests-marklogic-spark
 
 services:
 
@@ -27,31 +27,3 @@ services:
       - "8000-8002:8000-8002"
       - "8015:8015"
       - "8016:8016"
-
-  # Copied from https://docs.sonarsource.com/sonarqube/latest/setup-and-upgrade/install-the-server/#example-docker-compose-configuration .
-  sonarqube:
-    image: sonarqube:lts-community
-    depends_on:
-      - postgres
-    environment:
-      SONAR_JDBC_URL: jdbc:postgresql://postgres:5432/sonar
-      SONAR_JDBC_USERNAME: sonar
-      SONAR_JDBC_PASSWORD: sonar
-    volumes:
-      - sonarqube_data:/opt/sonarqube/data
-    ports:
-      - "9000:9000"
-
-  postgres:
-    image: postgres:15
-    environment:
-      POSTGRES_USER: sonar
-      POSTGRES_PASSWORD: sonar
-    volumes:
-      - postgresql:/var/lib/postgresql
-      - postgresql_data:/var/lib/postgresql/data
-
-volumes:
-  sonarqube_data:
-  postgresql:
-  postgresql_data:

docs/getting-started/jupyter.md

@@ -32,15 +32,15 @@ connector and also to initialize Spark:
 
 ```
 import os
-os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars "/path/to/marklogic-spark-connector-2.5.1.jar" pyspark-shell'
+os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars "/path/to/marklogic-spark-connector-2.6.0.jar" pyspark-shell'
 
 from pyspark.sql import SparkSession
 spark = SparkSession.builder.master("local[*]").appName('My Notebook').getOrCreate()
 spark.sparkContext.setLogLevel("WARN")
 spark
 ```
 
-The path of `/path/to/marklogic-spark-connector-2.5.1.jar` should be changed to match the location of the connector 
+The path of `/path/to/marklogic-spark-connector-2.6.0.jar` should be changed to match the location of the connector 
 jar on your filesystem. You are free to customize the `spark` variable in any manner you see fit as well. 
 
 Now that you have an initialized Spark session, you can run any of the examples found in the