Merge branch 'develop' into 11382-featured-items-unicode-filename #11382

Author: pdurbin

doc/release-notes/11415-app-tou.md

@@ -0,0 +1,3 @@
+### Application Terms of Use Available via API
+
+It's now possible to retrieve the Application Terms of Use (called General Terms of Use in the UI) via API. These are the terms users agree to when creating an account. See [the guides](https://dataverse-guide--11422.org.readthedocs.build/en/11422/api/native-api.html#get-application-terms-of-use-general-terms-of-use), #11415 and #11422.

doc/release-notes/9620-better-file-bundle-name.md

@@ -0,0 +1,2 @@
+The Data Access APIs that generate multi-file zipped bundles will offer file name suggestions based on the persistent identifiers (for example, `doi-10.70122-fk2-xxyyzz.zip`), instead of the fixed `dataverse_files.zip` as in prior versions.
+See the Data Access API guide for more info. 

doc/sphinx-guides/source/admin/metadatacustomization.rst

@@ -312,7 +312,7 @@ FieldType definitions
 |               | permitted, only a subset of HTML   |
 |               | tags will be rendered in the UI.   |
 |               | See the                            |
-|               | :ref:`supported-html-fields`       |
+|               | :ref:`supported-html-tags`         |
 |               | section of the Dataset + File      |
 |               | Management page in the User Guide. |
 +---------------+------------------------------------+

doc/sphinx-guides/source/api/dataaccess.rst

@@ -21,7 +21,7 @@ There are a number of reasons why not all of the files can be downloaded:
 - Some of the files are restricted and your API token doesn't have access (you will still get the unrestricted files).
 - The Dataverse installation has limited how large the zip bundle can be.
 
-In the curl example below, the flags ``-O`` and ``J`` are used. When there are no errors, this has the effect of saving the file as "dataverse_files.zip" (just like the web interface). The flags force errors to be downloaded as a file.
+In the curl example below, the flags ``-O`` and ``-J`` are used. When there are no errors, this has the effect of saving the file under the name suggested by Dataverse (which as of v6.7 will be based on the persistent identifier of the dataset and the latest version number, for example ``doi-10.70122-fk2-n2xgbj_1.1.zip``; in prior versions the file name was ``dataverse_files.zip`` in all cases). This mirrors the way the files are saved when downloaded in a browser. The flags also force error messages to be downloaded as a file.
 
 Please note that in addition to the files from dataset, an additional file call "MANIFEST.TXT" will be included in the zipped bundle. It has additional information about the files.
 
@@ -70,6 +70,8 @@ A curl example using a DOI (with version):
 
   curl -O -J -H "X-Dataverse-key:$API_TOKEN" $SERVER_URL/api/access/dataset/:persistentId/versions/$VERSION?persistentId=$PERSISTENT_ID
 
+Similarly to the API above, this will save the downloaded bundle under the name based on the persistent identifier and the version number, for example, ``doi-10.70122-fk2-n2xgbj_1.1.zip`` or ``doi-10.70122-fk2-n2xgbj_draft.zip``.
+
 The fully expanded example above (without environment variables) looks like this:
 
 .. code-block:: bash
@@ -173,7 +175,7 @@ Multiple File ("bundle") download
 
 Alternate Form: POST to ``/api/access/datafiles`` with a ``fileIds`` input field containing the same comma separated list of file ids. This is most useful when your list of files surpasses the allowed URL length (varies but can be ~2000 characters).  
 
-Returns the files listed, zipped. 
+Returns the files listed, zipped. As of v6.7 the name of the zipped bundle will be based on the persistent identifier of the parent dataset, for example, ``doi-10.70122-fk2-xxyyzz.zip``; in prior versions the file name was ``dataverse_files.zip`` in all cases).
 
 .. note:: If the request can only be completed partially - if only *some* of the requested files can be served (because of the permissions and/or size restrictions), the file MANIFEST.TXT included in the zipped bundle will have entries specifying the reasons the missing files could not be downloaded. IN THE FUTURE the API will return a 207 status code to indicate that the result was a partial success. (As of writing this - v.4.11 - this hasn't been implemented yet)
 

doc/sphinx-guides/source/api/native-api.rst

@@ -5544,6 +5544,28 @@ The fully expanded example above (without environment variables) looks like this
 
   curl "https://demo.dataverse.org/api/info/settings/:DatasetPublishPopupCustomText"
 
+.. _api-get-app-tou:
+
+Get Application Terms of Use (General Terms of Use)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the UI, Application Terms of Use is called "General Terms of Use" and can be seen when you sign up for an account. The terms come from the database setting :ref:`:ApplicationTermsOfUse`. If you have enabled :ref:`i18n` you can pass a two-character language code (e.g. "en") as the ``lang`` parameter.
+
+.. note:: See :ref:`curl-examples-and-environment-variables` if you are unfamiliar with the use of export below.
+
+.. code-block:: bash
+
+  export SERVER_URL=https://demo.dataverse.org
+  export LANG=en
+
+  curl "$SERVER_URL/api/info/applicationTermsOfUse?lang=$LANG"
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl "https://demo.dataverse.org/api/info/applicationTermsOfUse?lang=en"
+
 Get API Terms of Use URL
 ~~~~~~~~~~~~~~~~~~~~~~~~
 

doc/sphinx-guides/source/installation/config.rst

@@ -3921,22 +3921,31 @@ If you don't want the datafiles to be validated on publish, set:
 
 ``curl -X PUT -d 'false' http://localhost:8080/api/admin/settings/:FileValidationOnPublishEnabled``
 
+.. _:ApplicationTermsOfUse:
 
 :ApplicationTermsOfUse
 ++++++++++++++++++++++
 
-Upload an default language HTML file containing the Terms of Use to be displayed at sign up. Supported HTML tags are listed under the :doc:`/user/dataset-management` section of the User Guide.
+Application Terms of Use (called "General Terms of Use" in the UI) are shown to the user when they sign up for an account. Some HTML tags are supported. For a list, see :ref:`supported-html-tags` in the User Guide.
 
-``curl -X PUT -d@/tmp/apptou.html http://localhost:8080/api/admin/settings/:ApplicationTermsOfUse``
+You can set terms like this:
 
-To upload a language specific Terms of Use file,
+``curl -X PUT http://localhost:8080/api/admin/settings/:ApplicationTermsOfUse --upload-file /tmp/apptou.html``
 
-``curl -X PUT -d@/tmp/apptou_fr.html http://localhost:8080/api/admin/settings/:ApplicationTermsOfUse/lang/fr``
+To delete terms:
 
-To delete language specific option,
+``curl -X DELETE http://localhost:8080/api/admin/settings/:ApplicationTermsOfUse``
+
+If :ref:`i18n` is enabled, you can set the terms per language. To set terms for a specific language ("fr", for example):
+
+``curl -X PUT http://localhost:8080/api/admin/settings/:ApplicationTermsOfUse/lang/fr --upload-file /tmp/apptou_fr.html``
+
+To delete terms for a specific language ("fr", for example):
 
 ``curl -X DELETE http://localhost:8080/api/admin/settings/:ApplicationTermsOfUse/lang/fr``
 
+To retrieve the values, see :ref:`api-get-app-tou` in the API Guide.
+
 :ApplicationPrivacyPolicyUrl
 ++++++++++++++++++++++++++++
 

doc/sphinx-guides/source/user/dataset-management.rst

@@ -64,10 +64,10 @@ Adding a New Dataset
 
 Note: You can add additional metadata once you have completed the initial dataset creation by going to clicking the Edit button and selecting Metadata from the dropdown menu.
 
-.. _supported-html-fields:
+.. _supported-html-tags:
 
-Supported HTML Fields
----------------------
+Supported HTML Tags
+-------------------
 
 We currently only support the following HTML tags for any of our textbox metadata fields (i.e., Description) : <a>, <b>, <blockquote>, 
 <br>, <code>, <del>, <dd>, <dl>, <dt>, <em>, <hr>, <h1>-<h3>, <i>, <img>, <kbd>, <li>, <ol>, <p>, <pre>, <s>, <sup>, <sub>, 

src/main/java/edu/harvard/iq/dataverse/api/Access.java

@@ -169,6 +169,7 @@ public class Access extends AbstractApiBean {
     @Inject
     DataverseFeaturedItemServiceBean dataverseFeaturedItemServiceBean;
 
+    private static final String DEFAULT_BUNDLE_NAME = "dataverse_files.zip";
     //@EJB
 
     // TODO: 
@@ -643,7 +644,7 @@ public DownloadInstance downloadAuxiliaryFile(@Context ContainerRequestContext c
     public Response postDownloadDatafiles(@Context ContainerRequestContext crc, String fileIds, @QueryParam("gbrecs") boolean gbrecs, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) throws WebApplicationException {
 
 
-        return downloadDatafiles(getRequestUser(crc), fileIds, gbrecs, uriInfo, headers, response);
+        return downloadDatafiles(getRequestUser(crc), fileIds, gbrecs, uriInfo, headers, response, null);
     }
 
     @GET
@@ -664,7 +665,7 @@ public Response downloadAllFromLatest(@Context ContainerRequestContext crc, @Pat
                     // We don't want downloads from Draft versions to be counted, 
                     // so we are setting the gbrecs (aka "do not write guestbook response") 
                     // variable accordingly:
-                    return downloadDatafiles(getRequestUser(crc), fileIds, true, uriInfo, headers, response);
+                    return downloadDatafiles(getRequestUser(crc), fileIds, true, uriInfo, headers, response, "draft");
                 }
             }
 
@@ -685,7 +686,7 @@ public Response downloadAllFromLatest(@Context ContainerRequestContext crc, @Pat
             }
 
             String fileIds = getFileIdsAsCommaSeparated(latest.getFileMetadatas());
-            return downloadDatafiles(getRequestUser(crc), fileIds, gbrecs, uriInfo, headers, response);
+            return downloadDatafiles(getRequestUser(crc), fileIds, gbrecs, uriInfo, headers, response, latest.getFriendlyVersionNumber());
         } catch (WrappedResponse wr) {
             return wr.getResponse();
         }
@@ -735,7 +736,7 @@ public Command<DatasetVersion> handleLatestPublished() {
             if (dsv.isDraft()) {
                 gbrecs = true;
             }
-            return downloadDatafiles(getRequestUser(crc), fileIds, gbrecs, uriInfo, headers, response);
+            return downloadDatafiles(getRequestUser(crc), fileIds, gbrecs, uriInfo, headers, response, dsv.getFriendlyVersionNumber().toLowerCase());
         } catch (WrappedResponse wr) {
             return wr.getResponse();
         }
@@ -749,6 +750,24 @@ private static String getFileIdsAsCommaSeparated(List<FileMetadata> fileMetadata
         }
         return String.join(",", ids);
     }
+    
+    private String generateMultiFileBundleName(Dataset dataset, String versionTag) {
+        String bundleName = DEFAULT_BUNDLE_NAME;
+        
+        if (dataset != null && dataset.getGlobalId() != null) {
+            String protocol = dataset.getProtocol();
+            String authority = dataset.getAuthority().toLowerCase();
+            String identifier = dataset.getIdentifier().replace('/', '-').toLowerCase();
+                
+            if (versionTag != null) {
+                bundleName = protocol + "-" + authority + "-" + identifier + "_" + versionTag + ".zip"; 
+            } else {
+                bundleName = protocol + "-" + authority + "-" + identifier + ".zip";
+            }
+        }
+        
+        return bundleName;
+    }
 
     /*
      * API method for downloading zipped bundles of multiple files:
@@ -758,10 +777,10 @@ private static String getFileIdsAsCommaSeparated(List<FileMetadata> fileMetadata
     @Path("datafiles/{fileIds}")
     @Produces({"application/zip"})
     public Response datafiles(@Context ContainerRequestContext crc, @PathParam("fileIds") String fileIds, @QueryParam("gbrecs") boolean gbrecs, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) throws WebApplicationException {
-        return downloadDatafiles(getRequestUser(crc), fileIds, gbrecs, uriInfo, headers, response);
+        return downloadDatafiles(getRequestUser(crc), fileIds, gbrecs, uriInfo, headers, response, null);
     }
 
-    private Response downloadDatafiles(User user, String rawFileIds, boolean donotwriteGBResponse, UriInfo uriInfo, HttpHeaders headers, HttpServletResponse response) throws WebApplicationException /* throws NotFoundException, ServiceUnavailableException, PermissionDeniedException, AuthorizationRequiredException*/ {
+    private Response downloadDatafiles(User user, String rawFileIds, boolean donotwriteGBResponse, UriInfo uriInfo, HttpHeaders headers, HttpServletResponse response, String versionTag) throws WebApplicationException /* throws NotFoundException, ServiceUnavailableException, PermissionDeniedException, AuthorizationRequiredException*/ {
         final long zipDownloadSizeLimit = systemConfig.getZipDownloadLimit();
 
         logger.fine("setting zip download size limit to " + zipDownloadSizeLimit + " bytes.");
@@ -852,8 +871,9 @@ public void write(OutputStream os) throws IOException,
                                         // to produce some output.
                                         zipper = new DataFileZipper(os);
                                         zipper.setFileManifest(fileManifest);
-                                        response.setHeader("Content-disposition", "attachment; filename=\"dataverse_files.zip\"");
-                                        response.setHeader("Content-Type", "application/zip; name=\"dataverse_files.zip\"");
+                                        String bundleName = generateMultiFileBundleName(file.getOwner(), versionTag);
+                                        response.setHeader("Content-disposition", "attachment; filename=\"" + bundleName + "\"");
+                                        response.setHeader("Content-Type", "application/zip; name=\"" + bundleName + "\"");
                                     }
 
                                     long size = 0L;
@@ -960,8 +980,8 @@ public InputStream tempPreview(@PathParam("fileSystemId") String fileSystemId, @
 
     }*/
 
-    
-    
+        
+
     // TODO: Rather than only supporting looking up files by their database IDs, consider supporting persistent identifiers.
     @Path("fileCardImage/{fileId}")
     @GET

src/main/java/edu/harvard/iq/dataverse/api/Info.java

@@ -26,10 +26,13 @@
 import jakarta.json.JsonValue;
 import jakarta.ws.rs.GET;
 import jakarta.ws.rs.Path;
-import jakarta.ws.rs.PathParam;
+import jakarta.ws.rs.QueryParam;
 import jakarta.ws.rs.core.MediaType;
 import jakarta.ws.rs.core.Response;
 import org.eclipse.microprofile.openapi.annotations.Operation;
+import org.eclipse.microprofile.openapi.annotations.enums.SchemaType;
+import org.eclipse.microprofile.openapi.annotations.media.Schema;
+import org.eclipse.microprofile.openapi.annotations.parameters.Parameter;
 import org.eclipse.microprofile.openapi.annotations.responses.APIResponse;
 import org.eclipse.microprofile.openapi.annotations.tags.Tag;
 
@@ -78,6 +81,19 @@ public Response getServer() {
         return ok(JvmSettings.FQDN.lookup());
     }
 
+    @GET
+    @Path("applicationTermsOfUse")
+    @APIResponse(responseCode = "200",
+                 description = "Application Terms of Use (General Terms of Use) that must be agreed to at signup.")
+    public Response getApplicationTermsOfUse(
+            @Parameter(description = "Two-character language code.",
+                    required = false,
+                    example = "en",
+                    schema = @Schema(type = SchemaType.STRING))
+            @QueryParam("lang") String lang) {
+        return ok(systemConfig.getApplicationTermsOfUse(lang));
+    }
+
     @GET
     @Path("apiTermsOfUse")
     public Response getTermsOfUse() {

src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java

@@ -428,15 +428,24 @@ public boolean isThumbnailGenerationDisabledForPDF() {
     }
 
     public String getApplicationTermsOfUse() {
-        String language = BundleUtil.getCurrentLocale().getLanguage();
+        return getApplicationTermsOfUse(null);
+    }
+
+    public String getApplicationTermsOfUse(String languageIn) {
+        String language = null;
+        if (languageIn != null) {
+            language = languageIn;
+        } else {
+            language = BundleUtil.getCurrentLocale().getLanguage();
+        }
         String saneDefaultForAppTermsOfUse = BundleUtil.getStringFromBundle("system.app.terms");
-        // Get the value for the defaultLocale. IT will either be used as the return
+        // Get the value for the defaultLocale. It will either be used as the return
         // value, or as a better default than the saneDefaultForAppTermsOfUse if there
         // is no language-specific value
         String appTermsOfUse = settingsService.getValueForKey(SettingsServiceBean.Key.ApplicationTermsOfUse, saneDefaultForAppTermsOfUse);
-        //Now get the language-specific value if it exists
+        // Now get the language-specific value if it exists
         if (language != null && !language.equalsIgnoreCase(BundleUtil.getDefaultLocale().getLanguage())) {
-            appTermsOfUse = settingsService.getValueForKey(SettingsServiceBean.Key.ApplicationTermsOfUse, language,	appTermsOfUse);
+            appTermsOfUse = settingsService.getValueForKey(SettingsServiceBean.Key.ApplicationTermsOfUse, language, appTermsOfUse);
         }
         return appTermsOfUse;
     }