Merge pull request #309 from marklogic/release/2.4.0

Merge release/2.4.0 into master

Author: rjrudin

.env

@@ -1,3 +1,3 @@
 # Defines environment variables for docker-compose.
 # Can be overridden via e.g. `MARKLOGIC_TAG=latest-10.0 docker-compose up -d --build`.
-MARKLOGIC_TAG=11.2.0-centos-1.1.2
+MARKLOGIC_TAG=latest

CONTRIBUTING.md

@@ -1,28 +1,28 @@
 This guide covers how to develop and test this project. It assumes that you have cloned this repository to your local
 workstation.
 
-Due to the use of the Sonar plugin for Gradle, you must use Java 11 or higher for developing and testing the project. 
-The `build.gradle` file for this project ensures that the connector is built to run on Java 8 or higher. 
+You must use Java 11 or higher for developing, testing, and building this project. If you wish to use Sonar as 
+described below, you must use Java 17 or higher.
 
 # Setup
 
 To begin, you need to deploy the test application in this project to MarkLogic. You can do so either on your own 
-installation of MarkLogic, or you can use `docker-compose` to install MarkLogic, optionally as a 3-node cluster with 
+installation of MarkLogic, or you can use `docker compose` to install MarkLogic, optionally as a 3-node cluster with 
 a load balancer in front of it.
 
-## Installing MarkLogic with docker-compose
+## Installing MarkLogic with docker compose
 
-If you wish to use `docker-compose`, perform the following steps before deploying the test application.
+If you wish to use `docker compose`, perform the following steps before deploying the test application.
 
 1. [Install Docker](https://docs.docker.com/get-docker/).
 2. Ensure that you don't have a MarkLogic instance running locally (if you do, you may run into port conflicts in 
    the next step).
-3. Run `docker-compose up -d --build`.
+3. Run `docker compose up -d --build`.
 
 The above will result in a new MarkLogic instance with a single node. 
 
 Alternatively, if you would like to test against a 3-node MarkLogic cluster with a load balancer in front of it, 
-run `docker-compose -f docker-compose-3nodes.yaml up -d --build`.
+run `docker compose -f docker-compose-3nodes.yaml up -d --build`.
 
 ## Deploying the test application
 
@@ -45,8 +45,8 @@ To run the tests against the test application, run the following Gradle task:
 
 ## Generating code quality reports with SonarQube
 
-In order to use SonarQube, you must have used Docker to run this project's `docker-compose.yml` file and you must
-have the services in that file running.
+In order to use SonarQube, you must have used Docker to run this project's `docker-compose.yml` file, and you must
+have the services in that file running and you must use Java 17 to run the Gradle `sonar` task.
 
 To configure the SonarQube service, perform the following steps:
 
@@ -62,8 +62,8 @@ To configure the SonarQube service, perform the following steps:
 10. Add `systemProp.sonar.token=your token pasted here` to `gradle-local.properties` in the root of your project, creating
 that file if it does not exist yet.
 
-To run SonarQube, run the following Gradle tasks, which will run all the tests with code coverage and then generate
-a quality report with SonarQube:
+To run SonarQube, run the following Gradle tasks using Java 17, which will run all the tests with code coverage and 
+then generate a quality report with SonarQube:
 
     ./gradlew test sonar
 
@@ -83,20 +83,17 @@ you've introduced on the feature branch you're working on. You can then click on
 Note that if you only need results on code smells and vulnerabilities, you can repeatedly run `./gradlew sonar`
 without having to re-run the tests.
 
-Our Sonar instance is also configured to scan for dependency vulnerabilities 
-[via the dependency-check plugin](https://github.com/dependency-check/dependency-check-sonar-plugin). For more 
-information, see the `dependencyCheck` block in this project's `build.gradle` file. To include dependency check results,
-just run the following (it's not included by default when running the `sonar` task):
+You can also force Gradle to run `sonar` if any tests fail:
 
-    ./gradlew dependencyCheckAnalyze sonar
+    ./gradlew clean test sonar --continue
 
 ## Accessing MarkLogic logs in Grafana
 
 This project's `docker-compose-3nodes.yaml` file includes
 [Grafana, Loki, and promtail services](https://grafana.com/docs/loki/latest/clients/promtail/) for the primary reason of
 collecting MarkLogic log files and allowing them to be viewed and searched via Grafana.
 
-Once you have run `docker-compose`, you can access Grafana at http://localhost:3000 . Follow these instructions to
+Once you have run `docker compose`, you can access Grafana at http://localhost:3000 . Follow these instructions to
 access MarkLogic logging data:
 
 1. Click on the hamburger in the upper left hand corner and select "Explore", or simply go to
@@ -123,7 +120,7 @@ This will produce a single jar file for the connector in the `./build/libs` dire
 
 You can then launch PySpark with the connector available via:
 
-    pyspark --jars build/libs/marklogic-spark-connector-2.3-SNAPSHOT.jar
+    pyspark --jars build/libs/marklogic-spark-connector-2.4-SNAPSHOT.jar
 
 The below command is an example of loading data from the test application deployed via the instructions at the top of 
 this page. 
@@ -199,7 +196,7 @@ The Spark master GUI is at <http://localhost:8080>. You can use this to view det
 
 Now that you have a Spark cluster running, you just need to tell PySpark to connect to it:
 
-    pyspark --master spark://NYWHYC3G0W:7077 --jars build/libs/marklogic-spark-connector-2.3-SNAPSHOT.jar
+    pyspark --master spark://NYWHYC3G0W:7077 --jars build/libs/marklogic-spark-connector-2.4-SNAPSHOT.jar
 
 You can then run the same commands as shown in the PySpark section above. The Spark master GUI will allow you to 
 examine details of each of the commands that you run.
@@ -218,12 +215,12 @@ You will need the connector jar available, so run `./gradlew clean shadowJar` if
 You can then run a test Python program in this repository via the following (again, change the master address as 
 needed); note that you run this outside of PySpark, and `spark-submit` is available after having installed PySpark:
 
-    spark-submit --master spark://NYWHYC3G0W:7077 --jars build/libs/marklogic-spark-connector-2.3-SNAPSHOT.jar src/test/python/test_program.py
+    spark-submit --master spark://NYWHYC3G0W:7077 --jars build/libs/marklogic-spark-connector-2.4-SNAPSHOT.jar src/test/python/test_program.py
 
 You can also test a Java program. To do so, first move the `com.marklogic.spark.TestProgram` class from `src/test/java`
 to `src/main/java`. Then run `./gradlew clean shadowJar` to rebuild the connector jar. Then run the following:
 
-    spark-submit --master spark://NYWHYC3G0W:7077 --class com.marklogic.spark.TestProgram build/libs/marklogic-spark-connector-2.3-SNAPSHOT.jar
+    spark-submit --master spark://NYWHYC3G0W:7077 --class com.marklogic.spark.TestProgram build/libs/marklogic-spark-connector-2.4-SNAPSHOT.jar
 
 Be sure to move `TestProgram` back to `src/test/java` when you are done. 
 

Jenkinsfile

@@ -40,7 +40,7 @@ pipeline{
     buildDiscarder logRotator(artifactDaysToKeepStr: '7', artifactNumToKeepStr: '', daysToKeepStr: '30', numToKeepStr: '')
   }
   environment{
-    JAVA11_HOME_DIR="/home/builder/java/jdk-11.0.2"
+    JAVA17_HOME_DIR="/home/builder/java/jdk-17.0.2"
     GRADLE_DIR   =".gradle"
     DMC_USER     = credentials('MLBUILD_USER')
     DMC_PASSWORD = credentials('MLBUILD_PASSWORD')
@@ -61,9 +61,9 @@ pipeline{
             docker-compose down -v || true
             docker-compose up -d --build
           '''
-        runtests('JAVA11_HOME_DIR')
+        runtests('JAVA17_HOME_DIR')
         withSonarQubeEnv('SONAR_Progress') {
-          runSonarScan('JAVA11_HOME_DIR')
+          runSonarScan('JAVA17_HOME_DIR')
         }
       }
       post{
@@ -83,7 +83,7 @@ pipeline{
       }
       steps{
       	sh label:'publish', script: '''#!/bin/bash
-          export JAVA_HOME=$JAVA11_HOME_DIR
+          export JAVA_HOME=$JAVA17_HOME_DIR
           export GRADLE_USER_HOME=$WORKSPACE/$GRADLE_DIR
           export PATH=$GRADLE_USER_HOME:$JAVA_HOME/bin:$PATH
           cp ~/.gradle/gradle.properties $GRADLE_USER_HOME;
@@ -110,7 +110,7 @@ pipeline{
                 docker-compose down -v || true
                 MARKLOGIC_TAG=latest-10.0 docker-compose up -d --build
             '''
-            runtests('JAVA11_HOME_DIR')
+            runtests('JAVA17_HOME_DIR')
       }
       post{
         always{

build.gradle

@@ -2,20 +2,17 @@ plugins {
   id 'java-library'
   id 'net.saliman.properties' version '1.5.2'
   id 'com.github.johnrengelman.shadow' version '8.1.1'
-  id "com.marklogic.ml-gradle" version "4.7.0"
+  id "com.marklogic.ml-gradle" version "5.0.0"
   id 'maven-publish'
-  id 'signing'
   id "jacoco"
-  id "org.sonarqube" version "4.4.1.3373"
-  id "org.owasp.dependencycheck" version "10.0.3"
+  id "org.sonarqube" version "5.1.0.4882"
 }
 
 group 'com.marklogic'
-version '2.3.1'
+version '2.4.0'
 
 java {
-  // To support reading RDF files, Apache Jena is used - but that requires Java 11. If we want to do a 2.2.0 release
-  // without requiring Java 11, we'll remove the support for reading RDF files along with the Jena dependency.
+  // To support reading RDF files, Apache Jena is used - but that requires Java 11.
   sourceCompatibility = 11
   targetCompatibility = 11
 }
@@ -55,8 +52,8 @@ dependencies {
     exclude group: "com.fasterxml.jackson.dataformat"
   }
 
-  // Required for converting JSON to XML. Using 2.14.2 to align with Spark 3.4.1.
-  shadowDependencies "com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.14.2"
+  // Required for converting JSON to XML. Using 2.15.2 to align with Spark 3.5.3.
+  shadowDependencies "com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.15.2"
 
   // Need this so that an OkHttpClientConfigurator can be created.
   shadowDependencies 'com.squareup.okhttp3:okhttp:4.12.0'
@@ -68,7 +65,7 @@ dependencies {
 
   shadowDependencies "org.jdom:jdom2:2.0.6.1"
 
-  testImplementation ('com.marklogic:ml-app-deployer:4.8.0') {
+  testImplementation ('com.marklogic:ml-app-deployer:5.0.0') {
     exclude group: "com.fasterxml.jackson.core"
     exclude group: "com.fasterxml.jackson.dataformat"
 
@@ -85,27 +82,18 @@ dependencies {
   }
 
   testImplementation "ch.qos.logback:logback-classic:1.3.14"
-  testImplementation "org.slf4j:jcl-over-slf4j:1.7.36"
+  testImplementation "org.slf4j:jcl-over-slf4j:2.0.13"
   testImplementation "org.skyscreamer:jsonassert:1.5.1"
 }
 
-// See https://jeremylong.github.io/DependencyCheck/dependency-check-gradle/configuration.html for more information.
-dependencyCheck {
-  // Need a JSON report to integrate with Sonar. And HTML is easier for humans to read.
-  formats = ["HTML", "JSON"]
-  // We don't include compileOnly since that includes Spark, and Spark and its dependencies are not actual dependencies
-  // of our connector.
-  scanConfigurations = ["shadowDependencies"]
-  suppressionFile = "config/dependency-check-suppressions.xml"
-}
-
 test {
   useJUnitPlatform()
   finalizedBy jacocoTestReport
   // Allows mlHost to override the value in gradle.properties, which the test plumbing will default to.
   environment "mlHost", mlHost
 }
 
+// See https://docs.gradle.org/current/userguide/jacoco_plugin.html .
 jacocoTestReport {
   dependsOn test
   reports {
@@ -117,8 +105,6 @@ sonar {
   properties {
     property "sonar.projectKey", "marklogic-spark"
     property "sonar.host.url", "http://localhost:9000"
-    // See https://github.com/dependency-check/dependency-check-sonar-plugin for more information.
-    property "sonar.dependencyCheck.jsonReportPath", "build/reports/dependency-check-report.json"
   }
 }
 
@@ -136,7 +122,8 @@ if (JavaVersion.current().isCompatibleWith(JavaVersion.VERSION_17)) {
     // for an explanation of why these are needed when running the tests on Java 17.
     jvmArgs = [
       '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED',
-      '--add-opens=java.base/sun.util.calendar=ALL-UNNAMED'
+      '--add-opens=java.base/sun.util.calendar=ALL-UNNAMED',
+      '--add-opens=java.base/sun.security.action=ALL-UNNAMED'
     ]
   }
 }
@@ -178,9 +165,6 @@ javadoc.options.addStringOption('Xdoclint:none', '-quiet')
 artifacts {
   archives javadocJar, sourcesJar
 }
-signing {
-  sign configurations.archives
-}
 
 publishing {
   publications {

docker-compose-3nodes.yaml

@@ -1,4 +1,3 @@
-version: '3.8'
 name: marklogic_spark_3nodes
 
 services:
@@ -30,7 +29,7 @@ services:
   # by this host. Note that each MarkLogic host has its 8000-8002 ports exposed externally so that the apps on those
   # ports can each be accessed if needed.
   bootstrap_3n:
-    image: "marklogicdb/marklogic-db:${MARKLOGIC_TAG}"
+    image: "progressofficial/marklogic-db:${MARKLOGIC_TAG}"
     platform: linux/amd64
     container_name: bootstrap_3n
     hostname: bootstrap_3n.local
@@ -50,7 +49,7 @@ services:
       - internal_net
 
   node2:
-    image: "marklogicdb/marklogic-db:${MARKLOGIC_TAG}"
+    image: "progressofficial/marklogic-db:${MARKLOGIC_TAG}"
     platform: linux/amd64
     container_name: node2
     hostname: node2.local
@@ -74,7 +73,7 @@ services:
       - internal_net
 
   node3:
-    image: "marklogicdb/marklogic-db:${MARKLOGIC_TAG}"
+    image: "progressofficial/marklogic-db:${MARKLOGIC_TAG}"
     platform: linux/amd64
     container_name: node3
     hostname: node3.local

docker-compose.yaml

@@ -4,7 +4,7 @@ name: marklogic_spark
 services:
 
   marklogic:
-    image: "marklogicdb/marklogic-db:${MARKLOGIC_TAG}"
+    image: "progressofficial/marklogic-db:${MARKLOGIC_TAG}"
     platform: linux/amd64
     environment:
       - MARKLOGIC_INIT=true
@@ -19,24 +19,20 @@ services:
 
   # Copied from https://docs.sonarsource.com/sonarqube/latest/setup-and-upgrade/install-the-server/#example-docker-compose-configuration .
   sonarqube:
-    # Using 10.2 to avoid requiring Java 17 for now.
-    image: sonarqube:10.2.1-community
+    image: sonarqube:10.6.0-community
     depends_on:
       - postgres
     environment:
       SONAR_JDBC_URL: jdbc:postgresql://postgres:5432/sonar
       SONAR_JDBC_USERNAME: sonar
       SONAR_JDBC_PASSWORD: sonar
     volumes:
-      - ./docker/sonarqube/data:/opt/sonarqube/data
-      - ./docker/sonarqube/logs:/opt/sonarqube/logs
-      # Allows for Sonar plugins to be installed by including plugin jar files in this directory.
-      - ./docker/sonarqube/extensions:/opt/sonarqube/extensions
+      - sonarqube_data:/opt/sonarqube/data
     ports:
       - "9000:9000"
 
   postgres:
-    image: postgres:12
+    image: postgres:15
     environment:
       POSTGRES_USER: sonar
       POSTGRES_PASSWORD: sonar
@@ -46,7 +42,5 @@ services:
 
 volumes:
   sonarqube_data:
-  sonarqube_extensions:
-  sonarqube_logs:
   postgresql:
   postgresql_data:

docker/sonarqube/extensions/plugins/sonar-dependency-check-plugin-4.0.1.jar

@@ -32,15 +32,15 @@ connector and also to initialize Spark:
 
 ```
 import os
-os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars "/path/to/marklogic-spark-connector-2.3.1.jar" pyspark-shell'
+os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars "/path/to/marklogic-spark-connector-2.4.0.jar" pyspark-shell'
 
 from pyspark.sql import SparkSession
 spark = SparkSession.builder.master("local[*]").appName('My Notebook').getOrCreate()
 spark.sparkContext.setLogLevel("WARN")
 spark
 ```
 
-The path of `/path/to/marklogic-spark-connector-2.3.1.jar` should be changed to match the location of the connector 
+The path of `/path/to/marklogic-spark-connector-2.4.0.jar` should be changed to match the location of the connector 
 jar on your filesystem. You are free to customize the `spark` variable in any manner you see fit as well. 
 
 Now that you have an initialized Spark session, you can run any of the examples found in the 

docs/getting-started/jupyter.md

@@ -29,7 +29,7 @@ shell by pressing `ctrl-D`.
 
 Run PySpark from the directory that you downloaded the connector to per the [setup instructions](setup.md):
 
-    pyspark --jars marklogic-spark-connector-2.3.1.jar
+    pyspark --jars marklogic-spark-connector-2.4.0.jar
 
 The `--jars` command line option is PySpark's method for utilizing Spark connectors. Each Spark environment should have
 a similar mechanism for including third party connectors; please see the documentation for your particular Spark

docs/getting-started/pyspark.md

@@ -31,10 +31,10 @@ have an instance of MarkLogic running, you can skip step 4 below, but ensure tha
 extracted directory contains valid connection properties for your instance of MarkLogic.
 
 1. From [this repository's Releases page](https://github.com/marklogic/marklogic-spark-connector/releases), select 
-   the latest release and download the `marklogic-spark-getting-started-2.3.1.zip` file.
+   the latest release and download the `marklogic-spark-getting-started-2.4.0.zip` file.
 2. Extract the contents of the downloaded zip file. 
 3. Open a terminal window and go to the directory created by extracting the zip file; the directory should have a 
-   name of "marklogic-spark-getting-started-2.3.1".
+   name of "marklogic-spark-getting-started-2.4.0".
 4. Run `docker-compose up -d` to start an instance of MarkLogic
 5. Ensure that the `./gradlew` file is executable; depending on your operating system, you may need to run
    `chmod 755 gradlew` to make the file executable.