Merge branch 'develop' into 11766-new-io.gdcc.dataverse-spi

Author: ofahimIQSS

doc/release-notes/11752-croissant-restricted.md

@@ -0,0 +1,13 @@
+- The optional Croissant exporter has been updated to 0.1.6 to prevent variable names, variable descriptions, and variable types from being exposed for restricted files. See https://github.com/gdcc/exporter-croissant/pull/20 and #11752.
+
+## Upgrade Instructions
+
+### Update Croissant exporter, if enabled, and reexport metadata
+
+If you have enabled the Croissant dataset metadata exporter, you should upgrade to version 0.1.6.
+
+- Stop Payara.
+- Delete the old Croissant exporter jar file. It will be located in the directory defined by the `dataverse.spi.exporters.directory` setting.
+- Download the updated Croissant jar from https://repo1.maven.org/maven2/io/gdcc/export/croissant/ and place it in the same directory.
+- Restart Payara.
+- Run reExportAll.

doc/release-notes/11772-update-dataset-terms-of-access-api.md

@@ -0,0 +1,12 @@
+## New Endpoint: `/datasets/{id}/access`
+
+A new endpoint has been implemented to manage dataset terms of access for restricted files.
+
+### Functionality
+- Updates the terms of access for a dataset by applying it to the draft version.
+- If no draft exists, a new one is automatically created.
+
+### Usage
+
+**Custom Terms of Access** – Provide a JSON body with the `customTermsOfAccess` object.
+    - All fields are optional **except** if there are restricted files in which case `fileAccessRequest` must be set to true or  `termsOfAccess` must be provided.

doc/release-notes/11793 - ext. vocab. improvements.md

@@ -0,0 +1 @@
+This version of Dataverse includes extensions of the Dataverse External Vocabulary mechanism (https://guides.dataverse.org/en/latest/admin/metadatacustomization.html#using-external-vocabulary-services) that improve Dataverse's ability to include metadata about vocabulary terms and external identifiers such as ORCID and ROR in it's metadata exports. More information on how to configure external vocabulary scripts to use this functionality can be found at https://github.com/gdcc/dataverse-external-vocab-support/blob/main/docs/readme.md and in the examples in the https://github.com/gdcc/dataverse-external-vocab-support repository.

doc/release-notes/11796-fix-list-dataverse-templates-api-permissions.md

@@ -0,0 +1,10 @@
+### Fixed: Permissions for `/{identifier}/templates` endpoint
+
+The `/{identifier}/templates` endpoint previously required **`editDataverse`** permissions to retrieve the list of dataverse templates.
+
+This has been corrected: the endpoint now requires **`addDataset`** permissions instead.
+
+**Impact:**
+- The endpoint now works in scenarios such as the **Create Dataset form** in the SPA UI, without needing unnecessary elevated permissions.  
+
+Related issues: #11796

doc/release-notes/11855-version-summaries-pagination-and-perfomance.md

@@ -0,0 +1,33 @@
+### Pagination for API Version Summaries
+
+We've added pagination support to the following API endpoints:
+
+- File Version Differences: api/files/{id}/versionDifferences
+
+- Dataset Version Summaries: api/datasets/:persistentId/versions/compareSummary
+
+You can now use two new query parameters to control the results:
+
+- **limit**: An integer specifying the maximum number of results to return per page.
+
+- **offset**: An integer specifying the number of results to skip before starting to return items. This is used to
+  navigate to different pages.
+
+### Performance enhancements for API Version Summaries
+
+In addition to adding pagination, we've significantly improved the performance of these endpoints by implementing more
+efficient database queries.
+
+These changes address performance bottlenecks that were previously encountered, especially with datasets or files
+containing a large number of versions.
+
+### Fixes for File Version Summaries API
+
+The implementation for file version summaries was unreliable, leading to exceptions and functional inconsistencies, as
+documented in issue #11561. This functionality has been reviewed and fixed to ensure correctness and stability.
+
+### Related issues and PRs
+
+- https://github.com/IQSS/dataverse/issues/11855
+- https://github.com/IQSS/dataverse/pull/11859
+- https://github.com/IQSS/dataverse/issues/11561

doc/sphinx-guides/source/api/native-api.rst

@@ -2143,14 +2143,26 @@ be available to users who have permission to view unpublished drafts. The api to
   export SERVER_URL=https://demo.dataverse.org
   export PERSISTENT_IDENTIFIER=doi:10.5072/FK2/BCCP9Z
 
-  curl -H "X-Dataverse-key: $API_TOKEN" -X PUT "$SERVER_URL/api/datasets/:persistentId/versions/compareSummary?persistentId=$PERSISTENT_IDENTIFIER"
+  curl -H "X-Dataverse-key: $API_TOKEN" -X GET "$SERVER_URL/api/datasets/:persistentId/versions/compareSummary?persistentId=$PERSISTENT_IDENTIFIER"
 
 The fully expanded example above (without environment variables) looks like this:
 
 .. code-block:: bash
 
-  curl -H "X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X PUT "https://demo.dataverse.org/api/datasets/:persistentId/versions/compareSummary?persistentId=doi:10.5072/FK2/BCCP9Z"
+  curl -H "X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X GET "https://demo.dataverse.org/api/datasets/:persistentId/versions/compareSummary?persistentId=doi:10.5072/FK2/BCCP9Z"
 
+You can control pagination of the results using the following optional query parameters.
+
+* ``limit``: The maximum number of version differences to return.
+* ``offset``: The number of version differences to skip from the beginning of the list. Used for retrieving subsequent pages of results.
+
+To aid in pagination the JSON response also includes the total number of rows (totalCount) available.
+
+For example, to get the second page of results, with 2 items per page, you would use ``limit=2`` and ``offset=2`` (skipping the first two results).
+
+.. code-block:: bash
+
+  curl -H "X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X GET "https://demo.dataverse.org/api/datasets/:persistentId/versions/compareSummary?persistentId=doi:10.5072/FK2/BCCP9Z&limit=2&offset=2"
 
 Update Metadata For a Dataset
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -2286,6 +2298,46 @@ For these deletes your JSON file must include an exact match of those dataset fi
 
 .. _publish-dataset-api:
 
+Update Dataset Terms of Access
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Updates the terms of access for the restricted files of a dataset by applying it to the draft version, or by creating a draft if none exists.
+
+
+To define custom terms of access, provide a JSON body with the following properties. All fields within ``customTermsOfAccess`` are optional, except if there are restricted files in your dataset then ``fileAccessRequest`` must be set to true or ``termsOfAccess`` must be provided:
+
+.. code-block:: json
+
+  [
+    {
+      "customTermsOfAccess": {
+        "fileAccessRequest": true,
+        "termsOfAccess": "Your terms of access for restricted files",
+        "dataAccessPlace": "Your data access place",
+        "originalArchive": "Your original archive",
+        "availabilityStatus": "Your availability status",
+        "contactForAccess": "Your contact for access",
+        "sizeOfCollection": "Your size of collection",
+        "studyCompletion": "Your study completion"
+      }
+    }
+  ]
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export SERVER_URL=https://demo.dataverse.org
+  export ID=3
+  export FILE_PATH=access.json
+
+  curl -H "X-Dataverse-key:$API_TOKEN" -X PUT "$SERVER_URL/api/datasets/$ID/access" -H "Content-type:application/json" --upload-file $FILE_PATH
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X PUT "https://demo.dataverse.org/api/datasets/3/access" -H "Content-type:application/json" --upload-file access.json
+
 Publish a Dataset
 ~~~~~~~~~~~~~~~~~
 
@@ -4322,8 +4374,21 @@ The fully expanded example above (without environment variables) looks like this
 
 .. code-block:: bash
 
-  curl  -X GET "https://demo.dataverse.org/api/files/1234/versionDifferences"
-  curl  -X GET "https://demo.dataverse.org/api/files/:persistentId/versionDifferences?persistentId=doi:10.5072/FK2/J8SJZB"
+  curl -H "X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X GET "https://demo.dataverse.org/api/files/1234/versionDifferences"
+  curl -H "X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X GET "https://demo.dataverse.org/api/files/:persistentId/versionDifferences?persistentId=doi:10.5072/FK2/J8SJZB"
+
+You can control pagination of the results using the following optional query parameters.
+
+* ``limit``: The maximum number of version differences to return.
+* ``offset``: The number of version differences to skip from the beginning of the list. Used for retrieving subsequent pages of results.
+
+To aid in pagination the JSON response also includes the total number of rows (totalCount) available.
+
+For example, to get the second page of results, with 2 items per page, you would use ``limit=2`` and ``offset=2`` (skipping the first two results).
+
+.. code-block:: bash
+
+  curl -H "X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X GET "https://demo.dataverse.org/api/files/1234/versionDifferences?limit=2&offset=2"
 
 Adding Files
 ~~~~~~~~~~~~

src/main/java/edu/harvard/iq/dataverse/DataFileServiceBean.java

@@ -28,18 +28,18 @@
 import java.util.Map;
 import java.util.Set;
 import java.util.UUID;
+import java.util.function.Function;
 import java.util.logging.Level;
 import java.util.logging.Logger;
+import java.util.stream.Collectors;
+
 import jakarta.ejb.EJB;
 import jakarta.ejb.Stateless;
 import jakarta.ejb.TransactionAttribute;
 import jakarta.ejb.TransactionAttributeType;
 import jakarta.inject.Named;
-import jakarta.persistence.EntityManager;
-import jakarta.persistence.NoResultException;
-import jakarta.persistence.PersistenceContext;
-import jakarta.persistence.Query;
-import jakarta.persistence.TypedQuery;
+import jakarta.persistence.*;
+import jakarta.persistence.criteria.*;
 
 /**
  *
@@ -376,6 +376,133 @@ public FileMetadata findFileMetadataByDatasetVersionIdAndDataFileId(Long dataset
         }
     }
 
+    /**
+     * Finds the complete history of a file's presence across all dataset versions.
+     * <p>
+     * This method returns a {@link VersionedFileMetadata} entry for every version
+     * of the specified dataset. If a version does not contain the file, the
+     * {@code fileMetadata} field in the corresponding DTO will be {@code null}.
+     * It correctly handles file replacements by searching for all files sharing the
+     * same {@code rootDataFileId}.
+     *
+     * @param datasetId                  The ID of the parent dataset.
+     * @param dataFile                   The DataFile entity to find the history for.
+     * @param canViewUnpublishedVersions A boolean indicating if the user has permission to view non-released versions.
+     * @param limit                      (Optional) The maximum number of results to return.
+     * @param offset                     (Optional) The starting point of the result list.
+     * @return A chronologically sorted, paginated list of the file's version history, including versions where the file is absent.
+     */
+    public List<VersionedFileMetadata> findFileMetadataHistory(Long datasetId,
+                                                               DataFile dataFile,
+                                                               boolean canViewUnpublishedVersions,
+                                                               Integer limit,
+                                                               Integer offset) {
+        if (dataFile == null) {
+            return Collections.emptyList();
+        }
+
+        // Query 1: Get the paginated list of relevant DatasetVersions
+        CriteriaBuilder cb = em.getCriteriaBuilder();
+        CriteriaQuery<DatasetVersion> versionQuery = cb.createQuery(DatasetVersion.class);
+        Root<DatasetVersion> versionRoot = versionQuery.from(DatasetVersion.class);
+
+        List<Predicate> versionPredicates = new ArrayList<>();
+        versionPredicates.add(cb.equal(versionRoot.join("dataset").get("id"), datasetId));
+        if (!canViewUnpublishedVersions) {
+            versionPredicates.add(versionRoot.get("versionState").in(
+                    VersionState.RELEASED, VersionState.DEACCESSIONED));
+        }
+        versionQuery.where(versionPredicates.toArray(new Predicate[0]));
+        versionQuery.orderBy(
+                cb.desc(versionRoot.get("versionNumber")),
+                cb.desc(versionRoot.get("minorVersionNumber"))
+        );
+
+        TypedQuery<DatasetVersion> typedVersionQuery = em.createQuery(versionQuery);
+        if (limit != null) {
+            typedVersionQuery.setMaxResults(limit);
+        }
+        if (offset != null) {
+            typedVersionQuery.setFirstResult(offset);
+        }
+        List<DatasetVersion> datasetVersions = typedVersionQuery.getResultList();
+
+        if (datasetVersions.isEmpty()) {
+            return Collections.emptyList();
+        }
+
+        // Query 2: Get all FileMetadata for this file's history in this dataset
+        CriteriaQuery<FileMetadata> fmQuery = cb.createQuery(FileMetadata.class);
+        Root<FileMetadata> fmRoot = fmQuery.from(FileMetadata.class);
+
+        List<Predicate> fmPredicates = new ArrayList<>();
+        fmPredicates.add(cb.equal(fmRoot.get("datasetVersion").get("dataset").get("id"), datasetId));
+
+        // Find the file by its entire lineage
+        if (dataFile.getRootDataFileId() < 0) {
+            fmPredicates.add(cb.equal(fmRoot.get("dataFile").get("id"), dataFile.getId()));
+        } else {
+            fmPredicates.add(cb.equal(fmRoot.get("dataFile").get("rootDataFileId"), dataFile.getRootDataFileId()));
+        }
+        fmQuery.where(fmPredicates.toArray(new Predicate[0]));
+
+        List<FileMetadata> fileHistory = em.createQuery(fmQuery).getResultList();
+
+        // Combine results
+        Map<Long, FileMetadata> fmMap = fileHistory.stream()
+                .collect(Collectors.toMap(
+                        fm -> fm.getDatasetVersion().getId(),
+                        Function.identity()
+                ));
+
+        // Create the final list, looking up the FileMetadata for each version
+        return datasetVersions.stream()
+                .map(version -> new VersionedFileMetadata(
+                        version,
+                        fmMap.get(version.getId()) // This will be null if no entry exists for that version ID
+                ))
+                .collect(Collectors.toList());
+    }
+
+    /**
+     * Finds the FileMetadata for a given file in the version immediately preceding a specified version.
+     *
+     * @param fileMetadata   The FileMetadata instance from the current version, used to identify the file's lineage.
+     * @return The FileMetadata from the immediately prior version, or {@code null} if this is the first version of the file.
+     */
+    public FileMetadata getPreviousFileMetadata(FileMetadata fileMetadata) {
+        if (fileMetadata == null || fileMetadata.getDataFile() == null) {
+            return null;
+        }
+
+        // 1. Get the ID of the file that was replaced.
+        Long previousId = fileMetadata.getDataFile().getPreviousDataFileId();
+
+        // If there's no previous ID, this is the first version of the file.
+        if (previousId == null) {
+            return null;
+        }
+
+        CriteriaBuilder cb = em.getCriteriaBuilder();
+        CriteriaQuery<FileMetadata> cq = cb.createQuery(FileMetadata.class);
+        Root<FileMetadata> fileMetadataRoot = cq.from(FileMetadata.class);
+
+        // 2. Join FileMetadata to DataFile to access the ID.
+        Join<FileMetadata, DataFile> dataFileJoin = fileMetadataRoot.join("dataFile");
+
+        // 3. Find the FileMetadata whose DataFile ID matches the previousId.
+        cq.where(cb.equal(dataFileJoin.get("id"), previousId));
+
+        // --- Execution ---
+        TypedQuery<FileMetadata> query = em.createQuery(cq);
+        try {
+            return query.getSingleResult();
+        } catch (NoResultException e) {
+            // If no result is found, return null.
+            return null;
+        }
+    }
+
     public FileMetadata findMostRecentVersionFileIsIn(DataFile file) {
         if (file == null) {
             return null;

src/main/java/edu/harvard/iq/dataverse/DatasetFieldServiceBean.java

@@ -764,13 +764,24 @@ Object processPathSegment(int index, String[] pathParts, JsonValue curPath, Stri
                                 JsonValue val = jo.get(keyVal[0]);
                                 if (val != null) {
                                     if (val.getValueType().equals(ValueType.STRING)) {
+                                        //Match a string value
                                         if (((JsonString) val).getString().equals(expected)) {
                                             logger.fine("Found: " + jo);
                                             curPath = jo;
                                             return processPathSegment(index + 1, pathParts, curPath, termUri);
                                         }
-                                    } else {
-                                        logger.warning("Expected a string value for " + keyVal[0] + " but found: " + val.getValueType());
+                                    } else if (val.getValueType() == JsonValue.ValueType.ARRAY) {
+                                        // Match one string in an array
+                                        JsonArray jsonArray = (JsonArray) val;
+                                        for (JsonValue arrayVal : jsonArray) {
+                                            if (arrayVal.getValueType() == JsonValue.ValueType.STRING) {
+                                                if (((JsonString) arrayVal).getString().equals(expected)) {
+                                                    logger.fine("Found match in array: " + jo.toString());
+                                                    curPath = jo;
+                                                    return processPathSegment(index + 1, pathParts, curPath, termUri);
+                                                }
+                                            }
+                                        }
                                     }
                                 }
                             }

src/main/java/edu/harvard/iq/dataverse/DatasetVersion.java

@@ -73,7 +73,11 @@
     @NamedQuery(name = "DatasetVersion.findById", 
                 query = "SELECT o FROM DatasetVersion o LEFT JOIN FETCH o.fileMetadatas WHERE o.id=:id"), 
     @NamedQuery(name = "DatasetVersion.findByDataset",
-                query = "SELECT o FROM DatasetVersion o WHERE o.dataset.id=:datasetId ORDER BY o.versionNumber DESC, o.minorVersionNumber DESC"), 
+                query = "SELECT o FROM DatasetVersion o WHERE o.dataset.id=:datasetId ORDER BY o.versionNumber DESC, o.minorVersionNumber DESC"),
+    @NamedQuery(name = "DatasetVersion.findByDesiredStatesAndDataset",
+            query = "SELECT o FROM DatasetVersion o " +
+                    "WHERE o.dataset.id = :datasetId AND o.versionState IN :states " +
+                    "ORDER BY o.versionNumber DESC, o.minorVersionNumber DESC"),
     @NamedQuery(name = "DatasetVersion.findReleasedByDataset",
                 query = "SELECT o FROM DatasetVersion o WHERE o.dataset.id=:datasetId AND o.versionState=edu.harvard.iq.dataverse.DatasetVersion.VersionState.RELEASED ORDER BY o.versionNumber DESC, o.minorVersionNumber DESC")/*,
     @NamedQuery(name = "DatasetVersion.findVersionElements",