Merge pull request #778 from marklogic/release/6.0.0

MLE-22591 Merge release/6.0.0 into master

Author: rjrudin

.env

@@ -0,0 +1,2 @@
+MARKLOGIC_IMAGE=ml-docker-db-dev-tierpoint.bed-artifactory.bedford.progress.com/marklogic/marklogic-server-ubi-rootless:latest-12
+MARKLOGIC_LOGS_PATH=./docker/marklogic/logs

.gitignore

@@ -16,3 +16,5 @@ out
 gradle-local.properties
 snyk
 .vscode
+docker
+dep.txt

CODEOWNERS

@@ -2,4 +2,4 @@
 # Each line is a file pattern followed by one or more owners.
 
 # These owners will be the default owners for everything in the repo.
-* @anu3990 @billfarber @rjrudin
+* @anu3990 @billfarber @rjrudin @stevebio

CONTRIBUTING.md

@@ -2,21 +2,9 @@
 
 Here's how to get started with making contributions to this project:
 
-1. Install a JDK (Java Development Kit) version 9 or higher. (If you're working with ml-gradle 2.x or 3.x, you only need Java 8.)
+1. Install a JDK (Java Development Kit) version 8 or higher.
 1. Clone this repository.
-1. Assuming you want to use Eclipse for development, run "gradlew eclipse" (or "gradle eclipse" if you have Gradle installed locally). This will generate Eclipse project files.
-1. Import your repository folder as a new project in Eclipse. Everything should compile successfully and you should be good to go.
-
-To make contributions, check out the issues in the project and follow the below, subject-to-change process:
-
-1. For a given issue, create a local branch with the name of the issue (I'll use issue #9 as an example): git checkout issue-9
-1. Track the local branch to a remote one: git push -u origin issue-9
-1. Do your work on that local feature branch, ideally making small commits. Push to remote as often as you'd like. 
-1. When you're ready, submit a pull request from your feature branch for review. 
-
-Currently, I don't have any tests for ml-gradle, as most of the interesting functionality is in ml-app-deployer, where
-there are many automated tests. I also haven't found an easy way to test tasks or the MarkLogicPlugin class. So for the
-time being, don't worry about automated tests. 
+1. Import your repository folder as a new project into your IDE. Everything should compile successfully and you should be good to go.
 
 ## Formatting
 
@@ -25,14 +13,17 @@ either install that or follow the guidelines set in the file.
 
 ## How to test ml-gradle changes
 
-This part is a little tricky - in order to really test ml-gradle, you need to build it and publish it to your local 
-Maven repository (~/.m2) and then reference it in a separate Gradle file. Fortunately, the ./examples directory in this
-project has many projects that you can use for testing out your change to ml-gradle. Here's how to go about doing so:
+In order to test ml-gradle as a Gradle plugin, you need to build it and publish it to your local 
+Maven repository (~/.m2) and then reference it in a separate Gradle file. The `./examples` directory in this
+project has many projects that you can use for testing out your change to ml-gradle, with the 
+`./examples/local-testing-project` being intended for testing local changes to ml-gradle. 
 
-1. Make your changes to the ml-gradle source code
-1. Run "gradle -Pversion=issue-number publishToMavenLocal" (the version can be anything you want)
+To build and publish ml-gradle locally, perform the following steps:
 
-Then pick a project under ./examples, and replace the "plugins" block with the following code:
+1. Make your changes to the ml-gradle source code.
+1. Run `gradle publishToMavenLocal`.
+
+Then pick a project under `./examples`, and replace the `plugins` block with the following code:
 
     buildscript {
       repositories {
@@ -45,15 +36,38 @@ Then pick a project under ./examples, and replace the "plugins" block with the f
     }
     apply plugin: "com.marklogic.ml-gradle"
 
-The above unfortunately has to be done because while the "plugins" DSL in Gradle is very concise, 
-it can't be used for finding plugins in your local Maven repository. So we have to
-replace it with the much more verbose syntax. Of course, after you're done testing with this, change the Gradle file
-back to just use the plugins DSL.
+The above has to be done because while the "plugins" DSL in Gradle is very concise, it can't be used for finding 
+plugins in your local Maven repository.
 
 After making the above changes, when you run any Gradle task, Gradle will use the ml-gradle plugin that you published
 to your local Maven repository. 
 
 You can then repeat this process as often as you want - i.e. make more changes in the ml-gradle source, publish a new
 copy to your local Maven repo, and test it in the project you chose. 
 
-For a real example, see the "local-testing-project" project in the examples directory.
+## Automated Testing
+
+The automated tests for each of the three subprojects require access to a MarkLogic server, and the ml-javaclient-util &
+ml-gradle subprojects also require a test application deployed to it. To deploy that
+application in a new Docker container, run the following commands:
+
+```
+docker compose up -d --build
+cd test-app
+../gradlew -i mlDeploy
+cd ..
+```
+
+Once the test server and application are set up, you are ready to run the tests for each subproject. Those tests can be
+run in multiple ways, but the easiest is to simply use Gradle on the command-line as shown below.
+
+```
+./gradlew ml-javaclient-util:test
+./gradlew ml-app-deployer:test
+./gradlew ml-gradle:test
+```
+
+You can also run all tests via:
+
+    ./gradlew test
+

Jenkinsfile

@@ -1,74 +1,76 @@
 @Library('shared-libraries') _
 
-def runTests(String mlVersion, String javaVersion){
-  copyRPM 'Release',mlVersion
-  setUpML '$WORKSPACE/xdmp/src/Mark*.rpm'
-  sh label:'test', script: '''#!/bin/bash
-    export JAVA_HOME=$'''+javaVersion+'''
-    export GRADLE_USER_HOME=$WORKSPACE/$GRADLE_DIR
-    export PATH=$GRADLE_USER_HOME:$JAVA_HOME/bin:$PATH
-    cd $WORKSPACE/ml-gradle/ml-javaclient-util-test-app
-    echo "mlPassword=admin" > gradle-local.properties
-    ../gradlew -i :mlDeploy
-    cd $WORKSPACE/ml-gradle/
-    ./gradlew ml-javaclient-util:test || true
-    ./gradlew ml-app-deployer:test || true
-    ./gradlew ml-gradle:test || true
+def setupDockerMarkLogic(String image) {
+	sh label:'mlsetup', script: '''#!/bin/bash
+	echo "Removing any running MarkLogic server and clean up MarkLogic data directory"
+	sudo /usr/local/sbin/mladmin remove
+	sudo /usr/local/sbin/mladmin cleandata
+	cd ml-gradle
+	docker-compose down -v || true
+	docker volume prune -f
+	echo "Using image: "'''+image+'''
+	docker pull '''+image+'''
+	MARKLOGIC_IMAGE='''+image+''' MARKLOGIC_LOGS_PATH=/tmp/marklogic/logs docker compose up -d --build
+	echo "Waiting for MarkLogic server to initialize."
+	sleep 60s
   '''
-  junit '**/build/**/*.xml'
 }
 
-pipeline{
-  agent none
+def tearDownDocker() {
+	sh label:'tearDownDocker', script: '''#!/bin/bash
+		cd ml-gradle
+    docker compose down -v || true
+		docker volume prune -f
+		sudo rm -rf /tmp/marklogic/logs
+	'''
+}
+
+pipeline {
+  agent {label 'devExpLinuxPool'}
+
   options {
     checkoutToSubdirectory 'ml-gradle'
-    buildDiscarder logRotator(artifactDaysToKeepStr: '7', artifactNumToKeepStr: '', daysToKeepStr: '30', numToKeepStr: '')
+    buildDiscarder logRotator(artifactDaysToKeepStr: '7', artifactNumToKeepStr: '', daysToKeepStr: '7', numToKeepStr: '5')
   }
-  environment{
-    JAVA11_HOME_DIR="/home/builder/java/jdk-11.0.20"
-    JAVA8_HOME_DIR="/home/builder/java/openjdk-1.8.0-262"
-    JAVA17_HOME_DIR="/home/builder/java/jdk-17.0.2"
-    JAVA21_HOME_DIR="/home/builder/java/jdk-21.0.1"
+
+  environment {
+    JAVA_HOME_DIR="/home/builder/java/jdk-17.0.2"
     GRADLE_DIR   =".gradle"
     DMC_USER     = credentials('MLBUILD_USER')
     DMC_PASSWORD = credentials('MLBUILD_PASSWORD')
   }
-  stages{
-    stage('tests-java11'){
-      agent {label 'devExpLinuxPool'}
-      steps{
-        runTests('11.3.1','JAVA11_HOME_DIR')
-      }
-    }
-    stage('regressions'){
-      when {
-        anyOf {
-          branch 'dev'
-          branch 'master'
-        }
-        beforeAgent true
-      }
-      parallel{
-        stage('tests-java8'){
-          agent {label 'devExpLinuxPool'}
-          steps{
-            runTests('11.3.1','JAVA8_HOME_DIR')
-          }
-        }
-        stage('tests-java17'){
-          agent {label 'devExpLinuxPool'}
-          steps{
-            runTests('11.3.1','JAVA17_HOME_DIR')
-          }
-        }
-        stage('tests-java21'){
-          agent {label 'devExpLinuxPool'}
-          steps{
-            runTests('11.3.1','JAVA21_HOME_DIR')
-          }
-        }
+
+  stages {
+
+    stage('tests') {
+      steps {
+      	cleanupDocker()
+      	setupDockerMarkLogic("ml-docker-db-dev-tierpoint.bed-artifactory.bedford.progress.com/marklogic/marklogic-server-ubi-rootless:latest-12")
+        sh label:'test', script: '''#!/bin/bash
+        	# 'set -e' causes the script to fail if any command fails.
+        	set -e
+          export JAVA_HOME=$JAVA_HOME_DIR
+          export GRADLE_USER_HOME=$WORKSPACE/$GRADLE_DIR
+          export PATH=$GRADLE_USER_HOME:$JAVA_HOME/bin:$PATH
+          cd ml-gradle
+          cd test-app
+          ../gradlew -i mlDeploy
+          cd ..
+          # Running these separately to decrease likelihood of connection resets.
+          ./gradlew ml-javaclient-util:test || true
+					./gradlew ml-app-deployer:test || true
+					./gradlew ml-gradle:test || true
+        '''
+        junit '**/build/**/*.xml'
       }
+      post {
+				always {
+					tearDownDocker()
+					cleanupDocker()
+				}
+			}
     }
+
     stage('publish'){
       agent {label 'devExpLinuxPool'}
 			when {
@@ -81,7 +83,7 @@ pipeline{
           export PATH=$GRADLE_USER_HOME:$JAVA_HOME/bin:$PATH
           cp ~/.gradle/gradle.properties $GRADLE_USER_HOME;
           cd ml-gradle
-           ./gradlew publish
+          ./gradlew publish
         '''
       }
     }