Merge branch 'develop' into codeql #9252

Author: pdurbin

.env

@@ -1,5 +1,5 @@
 APP_IMAGE=gdcc/dataverse:unstable
-POSTGRES_VERSION=16
+POSTGRES_VERSION=17
 DATAVERSE_DB_USER=dataverse
-SOLR_VERSION=9.3.0
+SOLR_VERSION=9.8.0
 SKIP_DEPLOY=0

.github/ISSUE_TEMPLATE/bug_report.md

@@ -14,7 +14,7 @@ Thank you for contributing to the Dataverse Project through the creation of a bu
 WARNING: If this is a security issue it should be reported privately to [email protected]
 
 More information on bug issues and contributions can be found in the "Contributing to Dataverse" page:
-https://github.com/IQSS/dataverse/blob/develop/CONTRIBUTING.md#bug-reportsissues
+https://guides.dataverse.org/en/latest/contributor/index.html
 
 Please fill out as much of the template as you can.
 Start below this comment section.
@@ -44,7 +44,6 @@ Start below this comment section.
 **Any related open or closed issues to this bug report?**
 
 
-
 **Screenshots:**
 
 No matter the issue, screenshots are always welcome.
@@ -53,3 +52,7 @@ To add a screenshot, please use one of the following formats and/or methods desc
 
 * https://help.github.com/en/articles/file-attachments-on-issues-and-pull-requests
 *
+
+
+**Are you thinking about creating a pull request for this issue?**
+Help is always welcome, is this bug something you or your organization plan to fix?

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,7 +1,7 @@
 ---
 name: Feature request
 about: Suggest an idea or new feature for the Dataverse software!
-title: 'Feature Request/Idea:'
+title: 'Feature Request:'
 labels: 'Type: Feature'
 assignees: ''
 
@@ -11,7 +11,7 @@ assignees: ''
 Thank you for contributing to the Dataverse Project through the creation of a feature request!
 
 More information on ideas/feature requests and contributions can be found in the "Contributing to Dataverse" page:
-https://github.com/IQSS/dataverse/blob/develop/CONTRIBUTING.md#ideasfeature-requests
+https://guides.dataverse.org/en/latest/contributor/index.html
 
 Please fill out as much of the template as you can.
 Start below this comment section.
@@ -34,3 +34,6 @@ Start below this comment section.
 
 
 **Any open or closed issues related to this feature request?**
+
+**Are you thinking about creating a pull request for this feature?**  
+Help is always welcome, is this feature something you or your organization plan to implement?

.github/ISSUE_TEMPLATE/idea_proposal.md

@@ -0,0 +1,40 @@
+---
+name: Idea proposal
+about: Propose a new idea for discussion to improve the Dataverse software!
+title: 'Suggestion:'
+labels: 'Type: Suggestion'
+assignees: ''
+
+---
+
+<!--
+Thank you for contributing to the Dataverse Project through the creation of a feature request!
+
+More information on ideas/feature requests and contributions can be found in the "Contributing to Dataverse" page:
+https://guides.dataverse.org/en/latest/contributor/index.html
+
+Please fill out as much of the template as you can.
+Start below this comment section.
+-->
+
+**Overview of the Suggestion**
+
+
+**What kind of user is the suggestion intended for?**
+(Example users roles: API User, Curator, Depositor, Guest, Superuser, Sysadmin)
+
+
+**What inspired this idea?**
+
+
+**What existing behavior do you want changed?**
+
+
+**Any brand new behavior do you want to add to Dataverse?**
+
+
+**Any open or closed issues related to this suggestion?**
+
+
+**Are you thinking about creating a pull request for this issue?**
+Help is always welcome, is this idea something you or your organization plan to implement?

.github/PULL_REQUEST_TEMPLATE.md

@@ -2,7 +2,7 @@
 
 **Which issue(s) this PR closes**:
 
-Closes #
+- Closes #
 
 **Special notes for your reviewer**:
 

.github/actions/setup-maven/action.yml

@@ -0,0 +1,37 @@
+---
+name: "Setup Maven and Caches"
+description: "Determine Java version and setup Maven, including necessary caches."
+inputs:
+  git-reference:
+    description: 'The git reference (branch/tag) to check out'
+    required: false
+    default: '${{ github.ref }}'
+  pom-paths:
+    description: "List of paths to Maven POM(s) for cache dependency setup"
+    required: false
+    default: 'pom.xml'
+runs:
+  using: composite
+  steps:
+    - name: Checkout repository
+      uses: actions/checkout@v4
+      with:
+        ref: ${{ inputs.git-reference }}
+    - name: Determine Java version by reading the Maven property
+      shell: bash
+      run: |
+        echo "JAVA_VERSION=$(grep '<target.java.version>' ${GITHUB_WORKSPACE}/modules/dataverse-parent/pom.xml | cut -f2 -d'>' | cut -f1 -d'<')" | tee -a ${GITHUB_ENV}
+    - name: Set up JDK ${{ env.JAVA_VERSION }}
+      id: setup-java
+      uses: actions/setup-java@v4
+      with:
+        java-version: ${{ env.JAVA_VERSION }}
+        distribution: 'temurin'
+        cache: 'maven'
+        cache-dependency-path: ${{ inputs.pom-paths }}
+    - name: Download common cache on branch cache miss
+      if: ${{ steps.setup-java.outputs.cache-hit != 'true' }}
+      uses: actions/cache/restore@v4
+      with:
+        key: dataverse-maven-cache
+        path: ~/.m2/repository

.github/dependabot.yml

@@ -0,0 +1,11 @@
+# Set update schedule for GitHub Actions
+# https://docs.github.com/en/code-security/supply-chain-security/keeping-your-dependencies-updated-automatically/keeping-your-actions-up-to-date-with-dependabot
+
+version: 2
+updates:
+
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      # Check for updates to GitHub Actions daily
+      interval: "daily"

.github/workflows/check_property_files.yml

@@ -0,0 +1,32 @@
+name: "Properties Check"
+on:
+    pull_request:
+        paths:
+            - "src/**/*.properties"
+            - "scripts/api/data/metadatablocks/*"
+jobs:
+    duplicate_keys:
+        name: Duplicate Keys
+        runs-on: ubuntu-latest
+        steps:
+            - uses: actions/checkout@v4
+            - name: Run duplicates detection script
+              shell: bash
+              run: tests/check_duplicate_properties.sh
+
+    metadata_blocks_properties:
+        name: Metadata Blocks Properties
+        runs-on: ubuntu-latest
+        steps:
+            - uses: actions/checkout@v4
+            - name: Setup GraalVM + Native Image
+              uses: graalvm/setup-graalvm@v1
+              with:
+                github-token: ${{ secrets.GITHUB_TOKEN }}
+                java-version: '21'
+                distribution: 'graalvm-community'
+            - name: Setup JBang
+              uses: jbangdev/setup-jbang@main
+            - name: Run metadata block properties verification script
+              shell: bash
+              run: tests/verify_mdb_properties.sh

.github/workflows/container_app_pr.yml

@@ -20,29 +20,29 @@ jobs:
         if: ${{ github.repository_owner == 'IQSS' }}
         steps:
             # Checkout the pull request code as when merged
-            - uses: actions/checkout@v3
+            - uses: actions/checkout@v4
               with:
                   ref: 'refs/pull/${{ github.event.client_payload.pull_request.number }}/merge'
-            - uses: actions/setup-java@v3
+            - uses: actions/setup-java@v4
               with:
                   java-version: "17"
                   distribution: 'adopt'
-            - uses: actions/cache@v3
+            - uses: actions/cache@v4
               with:
                   path: ~/.m2
                   key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }}
                   restore-keys: ${{ runner.os }}-m2
 
             # Note: Accessing, pushing tags etc. to GHCR will only succeed in upstream because secrets.
             - name: Login to Github Container Registry
-              uses: docker/login-action@v2
+              uses: docker/login-action@v3
               with:
                   registry: ghcr.io
                   username: ${{ secrets.GHCR_USERNAME }}
                   password: ${{ secrets.GHCR_TOKEN }}
 
             - name: Set up QEMU for multi-arch builds
-              uses: docker/setup-qemu-action@v2
+              uses: docker/setup-qemu-action@v3
 
             # Get the image tag from either the command or default to branch name (Not used for now)
             #-   name: Get the target tag name
@@ -87,7 +87,7 @@ jobs:
                       :ship: [See on GHCR](https://github.com/orgs/gdcc/packages/container). Use by referencing with full name as printed above, mind the registry name.
 
             # Leave a note when things have gone sideways
-            - uses: peter-evans/create-or-update-comment@v3
+            - uses: peter-evans/create-or-update-comment@v4
               if: ${{ failure() }}
               with:
                   issue-number: ${{ github.event.client_payload.pull_request.number }}

.github/workflows/container_app_push.yml

@@ -5,6 +5,12 @@ on:
     # We are deliberately *not* running on push events here to avoid double runs.
     # Instead, push events will trigger from the base image and maven unit tests via workflow_call.
     workflow_call:
+        inputs:
+            base-image-ref:
+                type: string
+                description: "Reference of the base image to build on in full qualified form [<registry>/]<namespace>/<repo>:<tag>"
+                required: false
+                default: "gdcc/base:unstable"
     pull_request:
         branches:
             - develop
@@ -16,7 +22,6 @@ on:
 
 env:
     IMAGE_TAG: unstable
-    BASE_IMAGE_TAG: unstable
     REGISTRY: "" # Empty means default to Docker Hub
     PLATFORMS: "linux/amd64,linux/arm64"
     MASTER_BRANCH_TAG: alpha
@@ -33,20 +38,24 @@ jobs:
         if: ${{ github.repository_owner == 'IQSS' }}
 
         steps:
-            - name: Checkout repository
-              uses: actions/checkout@v3
-
-            - name: Set up JDK
-              uses: actions/setup-java@v3
+            - name: Checkout and Setup Maven
+              uses: IQSS/dataverse/.github/actions/setup-maven@develop
               with:
-                  java-version: "17"
-                  distribution: temurin
-                  cache: maven
+                pom-paths: |
+                  pom.xml
+                  modules/container-configbaker/pom.xml
+                  modules/dataverse-parent/pom.xml
+
+            # TODO: Add a filter step here, that avoids building the image if this is a PR and there are other files touched than declared above.
+            #       Use https://github.com/dorny/paths-filter to solve this. This will ensure we do not run this twice if this workflow
+            #       will be triggered by the other workflows already (base image or java changes)
+            #       To become a part of #10618.
 
             - name: Build app and configbaker container image with local architecture and submodules (profile will skip tests)
               run: >
                   mvn -B -f modules/dataverse-parent
                   -P ct -pl edu.harvard.iq:dataverse -am
+                  $( [[ -n "${{ inputs.base-image-ref }}" ]] && echo "-Dbase.image=${{ inputs.base-image-ref }}" )
                   install
 
             # TODO: add smoke / integration testing here (add "-Pct -DskipIntegrationTests=false")
@@ -59,15 +68,15 @@ jobs:
         if: ${{ github.event_name != 'pull_request' && github.ref_name == 'develop' && github.repository_owner == 'IQSS' }}
         runs-on: ubuntu-latest
         steps:
-            - uses: actions/checkout@v3
-            - uses: peter-evans/dockerhub-description@v3
+            - uses: actions/checkout@v4
+            - uses: peter-evans/dockerhub-description@v4
               with:
                   username: ${{ secrets.DOCKERHUB_USERNAME }}
                   password: ${{ secrets.DOCKERHUB_TOKEN }}
                   repository: gdcc/dataverse
                   short-description: "Dataverse Application Container Image providing the executable"
                   readme-filepath: ./src/main/docker/README.md
-            - uses: peter-evans/dockerhub-description@v3
+            - uses: peter-evans/dockerhub-description@v4
               with:
                   username: ${{ secrets.DOCKERHUB_USERNAME }}
                   password: ${{ secrets.DOCKERHUB_TOKEN }}
@@ -106,29 +115,31 @@ jobs:
         if: needs.check-secrets.outputs.available == 'true' &&
             ( github.event_name != 'push' || ( github.event_name == 'push' && contains(fromJSON('["develop", "master"]'), github.ref_name)))
         steps:
-            - uses: actions/checkout@v3
-            - uses: actions/setup-java@v3
+            - name: Checkout and Setup Maven
+              uses: IQSS/dataverse/.github/actions/setup-maven@develop
               with:
-                  java-version: "17"
-                  distribution: temurin
+                pom-paths: |
+                  pom.xml
+                  modules/container-configbaker/pom.xml
+                  modules/dataverse-parent/pom.xml
 
             # Depending on context, we push to different targets. Login accordingly.
             - if: github.event_name != 'pull_request'
               name: Log in to Docker Hub registry
-              uses: docker/login-action@v2
+              uses: docker/login-action@v3
               with:
                   username: ${{ secrets.DOCKERHUB_USERNAME }}
                   password: ${{ secrets.DOCKERHUB_TOKEN }}
             - if: ${{ github.event_name == 'pull_request' }}
               name: Login to Github Container Registry
-              uses: docker/login-action@v2
+              uses: docker/login-action@v3
               with:
                   registry: ghcr.io
                   username: ${{ secrets.GHCR_USERNAME }}
                   password: ${{ secrets.GHCR_TOKEN }}
 
             - name: Set up QEMU for multi-arch builds
-              uses: docker/setup-qemu-action@v2
+              uses: docker/setup-qemu-action@v3
 
             - name: Re-set image tag based on branch (if master)
               if: ${{ github.ref_name == 'master' }}
@@ -146,11 +157,13 @@ jobs:
               run: >
                   mvn -B -f modules/dataverse-parent
                   -P ct -pl edu.harvard.iq:dataverse -am
+                  $( [[ -n "${{ inputs.base-image-ref }}" ]] && echo "-Dbase.image=${{ inputs.base-image-ref }}" )
                   install
             - name: Deploy multi-arch application and configbaker container image
               run: >
                   mvn 
-                  -Dapp.image.tag=${{ env.IMAGE_TAG }} -Dbase.image.tag=${{ env.BASE_IMAGE_TAG }}
+                  -Dapp.image.tag=${{ env.IMAGE_TAG }}
+                  $( [[ -n "${{ inputs.base-image-ref }}" ]] && echo "-Dbase.image=${{ inputs.base-image-ref }}" )
                   ${{ env.REGISTRY }} -Ddocker.platforms=${{ env.PLATFORMS }}
                   -P ct deploy