Merge branch 'develop' into 11634-api-get-available-file-categories

Author: sekmiller

doc/release-notes/11492-list-dataset-links.md

@@ -0,0 +1 @@
+The [API for listing the collections a dataset has been linked to](https://guides.dataverse.org/en/latest/admin/dataverses-datasets.html#list-collections-that-are-linked-from-a-dataset) (`api/datasets/$linked-dataset-id/links`) is no longer restricted to superusers. For unpublished datasets, users need the "View Unpublished Dataset" permission to access the API. Unpublished collections in the list require the "View Unpublished Dataverse" permission; otherwise, they are hidden.

doc/release-notes/11562-templates-api.md

@@ -0,0 +1,4 @@
+New endpoints have been implemented in the Dataverses API for the management of dataverse templates:
+
+- POST `/dataverses/{id}/templates`: Creates a template for a given Dataverse collection ``id``.
+- GET `/dataverses/{id}/templates`: Lists the templates for a given Dataverse collection ``id``.

doc/sphinx-guides/source/_static/api/dataverse-template.json

@@ -0,0 +1,27 @@
+{
+  "name": "Dataverse template",
+  "isDefault": true,
+  "fields": [
+    {
+      "typeName": "author",
+      "value": [
+        {
+          "authorName": {
+            "typeName": "authorName",
+            "value": "Belicheck, Bill"
+          },
+          "authorAffiliation": {
+            "typeName": "authorIdentifierScheme",
+            "value": "ORCID"
+          }
+        }
+      ]
+    }
+  ],
+  "instructions": [
+    {
+      "instructionField": "author",
+      "instructionText": "The author data"
+    }
+  ]
+}

doc/sphinx-guides/source/api/native-api.rst

@@ -1400,6 +1400,46 @@ The fully expanded example above (without environment variables) looks like this
 
   curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X GET "https://demo.dataverse.org/api/access/dataverseFeaturedItemImage/1"
 
+List Templates of a Collection
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Lists the templates for a given Dataverse collection ``id``:
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export SERVER_URL=https://demo.dataverse.org
+  export ID=1
+
+  curl -H "X-Dataverse-key:$API_TOKEN" -X GET "$SERVER_URL/api/dataverses/{ID}/templates"
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X GET "https://demo.dataverse.org/api/dataverses/1/templates"
+
+Create a Template for a Collection
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Creates a template for a given Dataverse collection ``id``.
+
+To create the template, you must send a JSON file. Your JSON file might look like :download:`dataverse-template.json <../_static/api/dataverse-template.json>` which you would send to the Dataverse installation like this:
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export SERVER_URL=https://demo.dataverse.org
+  export ID=1
+
+  curl -H "X-Dataverse-key: $API_TOKEN" -X POST "$SERVER_URL/api/dataverses/{ID}/templates" --upload-file dataverse-template.json
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl -H "X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X POST "https://demo.dataverse.org/api/dataverses/1/templates" --upload-file dataverse-template.json
+
 Datasets
 --------
 

doc/sphinx-guides/source/user/appendix.rst

@@ -23,7 +23,9 @@ Supported Metadata
 Detailed below are what metadata schemas we support for Citation and Domain Specific Metadata in the Dataverse Project:
 
 - Citation Metadata (`see .tsv <https://github.com/IQSS/dataverse/blob/master/scripts/api/data/metadatablocks/citation.tsv>`__): compliant with `DDI Lite <https://www.ddialliance.org/specification/ddi2.1/lite/index.html>`_, `DDI 2.5 Codebook <https://www.ddialliance.org/>`__, `DataCite 4.5 <https://schema.datacite.org/meta/kernel-4.5/>`__, and Dublin Core's `DCMI Metadata Terms <https://dublincore.org/documents/dcmi-terms/>`__ . Language field uses `ISO 639-1 <https://www.loc.gov/standards/iso639-2/php/English_list.php>`__ controlled vocabulary.
-- Geospatial Metadata (`see .tsv <https://github.com/IQSS/dataverse/blob/master/scripts/api/data/metadatablocks/geospatial.tsv>`__): compliant with `DDI Lite <https://www.ddialliance.org/specification/ddi2.1/lite/index.html>`_, `DDI 2.5 Codebook <https://www.ddialliance.org/>`__, `DataCite 4.5 <https://schema.datacite.org/meta/kernel-4.5/>`__, and Dublin Core. Country / Nation field uses `ISO 3166-1 <https://en.wikipedia.org/wiki/ISO_3166-1>`_ controlled vocabulary.
+- Geospatial Metadata (`see .tsv <https://github.com/IQSS/dataverse/blob/master/scripts/api/data/metadatablocks/geospatial.tsv>`__): compliant with `DDI Lite <https://www.ddialliance.org/specification/ddi2.1/lite/index.html>`_, `DDI 2.5 Codebook <https://www.ddialliance.org/>`__, `DataCite 4.5 <https://schema.datacite.org/meta/kernel-4.5/>`__, and Dublin Core. Country / Nation field uses `ISO 3166-1 <https://en.wikipedia.org/wiki/ISO_3166-1>`_ controlled vocabulary. 
+
+  - Please note that a new and improved Geospatial metadata block is being proposed at `<https://github.com/IQSS/dataverse/pull/11507>`__. We encourage you to try the block and give feedback in that pull request. 
 - Social Science & Humanities Metadata (`see .tsv <https://github.com/IQSS/dataverse/blob/master/scripts/api/data/metadatablocks/social_science.tsv>`__): compliant with `DDI Lite <https://www.ddialliance.org/specification/ddi2.1/lite/index.html>`_, `DDI 2.5 Codebook <https://www.ddialliance.org/>`__, and Dublin Core.
 - Astronomy and Astrophysics Metadata (`see .tsv <https://github.com/IQSS/dataverse/blob/master/scripts/api/data/metadatablocks/astrophysics.tsv>`__): These metadata elements can be mapped/exported to the International Virtual Observatory Alliance’s (IVOA) 
   `VOResource Schema format <https://www.ivoa.net/documents/latest/RM.html>`__ and is based on 

src/main/java/edu/harvard/iq/dataverse/EjbDataverseEngine.java

@@ -6,6 +6,7 @@
 import edu.harvard.iq.dataverse.authorization.AuthenticationServiceBean;
 import edu.harvard.iq.dataverse.authorization.providers.builtin.BuiltinUserServiceBean;
 import edu.harvard.iq.dataverse.dataverse.featured.DataverseFeaturedItemServiceBean;
+import edu.harvard.iq.dataverse.license.LicenseServiceBean;
 import edu.harvard.iq.dataverse.util.cache.CacheFactoryBean;
 import edu.harvard.iq.dataverse.engine.DataverseEngine;
 import edu.harvard.iq.dataverse.authorization.Permission;
@@ -192,6 +193,9 @@ public class EjbDataverseEngine {
     @EJB
     DataverseFeaturedItemServiceBean dataverseFeaturedItemServiceBean;
 
+    @EJB
+    LicenseServiceBean licenseServiceBean;
+
     @EJB
     DatasetFieldsValidator datasetFieldsValidator;
 
@@ -541,6 +545,11 @@ public DatasetFieldsValidator datasetFieldsValidator() {
                     return datasetFieldsValidator;
                 }
 
+                @Override
+                public LicenseServiceBean licenses() {
+                    return licenseServiceBean;
+                }
+
                 @Override
                 public StorageUseServiceBean storageUse() {
                     return storageUseService;

src/main/java/edu/harvard/iq/dataverse/Template.java

@@ -137,7 +137,11 @@ public List<DatasetField> getDatasetFields() {
 
     @Transient
     private Map<String, String> instructionsMap = null;
-    
+
+    public void setInstructionsMap(Map<String, String> instructionsMap) {
+        this.instructionsMap = instructionsMap;
+    }
+
     @Transient
     private TreeMap<MetadataBlock, List<DatasetField>> metadataBlocksForView = new TreeMap<>();
     @Transient

src/main/java/edu/harvard/iq/dataverse/api/Datasets.java

@@ -2089,20 +2089,23 @@ public Response getCustomTermsTab(@PathParam("id") String id, @PathParam("versio
     public Response getLinks(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied ) {
         try {
             User u = getRequestUser(crc);
-            if (!u.isSuperuser()) {
-                return error(Response.Status.FORBIDDEN, "Not a superuser");
-            }
             Dataset dataset = findDatasetOrDie(idSupplied);
 
+            if (!dataset.isReleased() && !permissionService.hasPermissionsFor(u, dataset, EnumSet.of(Permission.ViewUnpublishedDataset))) {
+                return error(Response.Status.FORBIDDEN, "User is not allowed to list the link(s) of this dataset");
+            }
+
             long datasetId = dataset.getId();
             List<Dataverse> dvsThatLinkToThisDatasetId = dataverseSvc.findDataversesThatLinkToThisDatasetId(datasetId);
             JsonArrayBuilder dataversesThatLinkToThisDatasetIdBuilder = Json.createArrayBuilder();
             for (Dataverse dataverse : dvsThatLinkToThisDatasetId) {
-                JsonObjectBuilder datasetBuilder = Json.createObjectBuilder();
-                datasetBuilder.add("id", dataverse.getId());
-                datasetBuilder.add("alias", dataverse.getAlias());
-                datasetBuilder.add("displayName", dataverse.getDisplayName());
-                dataversesThatLinkToThisDatasetIdBuilder.add(datasetBuilder.build());
+                if (dataverse.isReleased() || this.permissionService.hasPermissionsFor(u, dataverse, EnumSet.of(Permission.ViewUnpublishedDataverse))) {
+                    JsonObjectBuilder datasetBuilder = Json.createObjectBuilder();
+                    datasetBuilder.add("id", dataverse.getId());
+                    datasetBuilder.add("alias", dataverse.getAlias());
+                    datasetBuilder.add("displayName", dataverse.getDisplayName());
+                    dataversesThatLinkToThisDatasetIdBuilder.add(datasetBuilder.build());
+                }
             }
             JsonObjectBuilder response = Json.createObjectBuilder();
             response.add("id", datasetId);

src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java

@@ -1931,4 +1931,34 @@ public Response deleteFeaturedItems(@Context ContainerRequestContext crc, @PathP
             return e.getResponse();
         }
     }
+
+    @GET
+    @AuthRequired
+    @Path("{identifier}/templates")
+    public Response getTemplates(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf) {
+        try {
+            Dataverse dataverse = findDataverseOrDie(dvIdtf);
+            return ok(jsonTemplates(execCommand(new ListDataverseTemplatesCommand(createDataverseRequest(getRequestUser(crc)), dataverse))));
+        } catch (WrappedResponse e) {
+            return e.getResponse();
+        }
+    }
+
+    @POST
+    @AuthRequired
+    @Path("{identifier}/templates")
+    public Response createTemplate(@Context ContainerRequestContext crc, String body, @PathParam("identifier") String dvIdtf) {
+        try {
+            Dataverse dataverse = findDataverseOrDie(dvIdtf);
+            NewTemplateDTO newTemplateDTO;
+            try {
+                newTemplateDTO = NewTemplateDTO.fromRequestBody(body, jsonParser());
+            } catch (JsonParseException ex) {
+                return error(Status.BAD_REQUEST, MessageFormat.format(BundleUtil.getStringFromBundle("dataverse.createTemplate.error.jsonParseMetadataFields"), ex.getMessage()));
+            }
+            return ok(jsonTemplate(execCommand(new CreateTemplateCommand(newTemplateDTO.toTemplate(), createDataverseRequest(getRequestUser(crc)), dataverse, true))));
+        } catch (WrappedResponse e) {
+            return e.getResponse();
+        }
+    }
 }

src/main/java/edu/harvard/iq/dataverse/api/dto/NewTemplateDTO.java

@@ -0,0 +1,75 @@
+package edu.harvard.iq.dataverse.api.dto;
+
+import edu.harvard.iq.dataverse.*;
+import edu.harvard.iq.dataverse.util.json.JsonParseException;
+import edu.harvard.iq.dataverse.util.json.JsonParser;
+import edu.harvard.iq.dataverse.util.json.JsonUtil;
+import jakarta.json.*;
+
+import java.sql.Timestamp;
+import java.util.*;
+
+public class NewTemplateDTO {
+
+    private String name;
+    private List<DatasetField> datasetFields;
+    private Map<String, String> instructionsMap;
+    private boolean isDefault;
+
+    public static NewTemplateDTO fromRequestBody(String requestBody, JsonParser jsonParser) throws JsonParseException {
+        NewTemplateDTO newTemplateDTO = new NewTemplateDTO();
+
+        JsonObject jsonObject = JsonUtil.getJsonObject(requestBody);
+
+        newTemplateDTO.name = jsonObject.getString("name");
+        newTemplateDTO.datasetFields = jsonParser.parseMultipleFields(jsonObject);
+        newTemplateDTO.instructionsMap = parseRequestBodyInstructionsMap(jsonObject);
+        newTemplateDTO.isDefault = jsonObject.getBoolean("isDefault", false);
+
+        return newTemplateDTO;
+    }
+
+    public Template toTemplate() {
+        Template template = new Template();
+
+        template.setDatasetFields(getDatasetFields());
+        template.setName(getName());
+        template.setInstructionsMap(getInstructionsMap());
+        template.updateInstructions();
+        template.setCreateTime(new Timestamp(new Date().getTime()));
+        template.setUsageCount(0L);
+
+        return template;
+    }
+
+    public String getName() {
+        return name;
+    }
+
+    public List<DatasetField> getDatasetFields() {
+        return datasetFields;
+    }
+
+    public Map<String, String> getInstructionsMap() {
+        return instructionsMap;
+    }
+
+    public boolean isDefault() {
+        return isDefault;
+    }
+
+    private static Map<String, String> parseRequestBodyInstructionsMap(JsonObject jsonObject) {
+        Map<String, String> instructionsMap = new HashMap<>();
+        JsonArray instructionsJsonArray = jsonObject.getJsonArray("instructions");
+        if (instructionsJsonArray == null) {
+            return null;
+        }
+        for (JsonObject instructionJsonObject : instructionsJsonArray.getValuesAs(JsonObject.class)) {
+            instructionsMap.put(
+                    instructionJsonObject.getString("instructionField"),
+                    instructionJsonObject.getString("instructionText")
+            );
+        }
+        return instructionsMap;
+    }
+}