Merge branch 'develop' into 11546-pid-separator

Author: sekmiller

conf/solr/schema.xml

@@ -157,7 +157,8 @@
     <field name="dvSubject" type="string" stored="true" indexed="true" multiValued="true"/>
 
     <field name="publicationStatus" type="string" stored="true" indexed="true" multiValued="true"/>
-    <field name="externalStatus" type="string" stored="true" indexed="true" multiValued="false"/>
+    <field name="curationStatus" type="string" stored="true" indexed="true" multiValued="false"/>
+    <field name="curationStatusCreateTime" type="pdate" indexed="true" stored="true"/>
     <field name="embargoEndDate" type="plong" stored="true" indexed="true" multiValued="false"/>
     <field name="retentionEndDate" type="plong" stored="true" indexed="true" multiValued="false"/>
 

doc/release-notes/11360-AwsSdkV2.x.md

@@ -0,0 +1,13 @@
+## Upgrade to AWS SDK v2 (for S3), v1 EOL in December 2025
+
+To support S3 storage, Dataverse uses the AWS SDK. We have upgraded to v2 of this SDK because v1 reaches End Of Life (EOL) in [December 2025](https://aws.amazon.com/fr/blogs/developer/announcing-end-of-support-for-aws-sdk-for-java-v1-x-on-december-31-2025/).
+
+As part of the upgrade, the payload-signing setting for S3 stores (`dataverse.files.<id>.payload-signing`) has been removed because it is no longer necessary. With the updated SDK, a payload signature will automatically be sent when required (and not sent when not required).
+
+Dataverse developers should note that LocalStack is used to test S3 and older versions appear to be incompatible. The development environment has been upgraded to LocalStack v2.3.2 to v4.2.0, which seems to work fine.
+
+See also #11073 and #11360.
+
+### Settings Removed
+
+- `dataverse.files.<id>.payload-signing`

doc/release-notes/9247-CurationStatusUpdates.md

@@ -0,0 +1,14 @@
+The External/Curation Status Label mechanism has been enhanced:
+
+- adding tracking of who creates the status label and when,
+- keeping a history of past statuses
+- updating the CSV report to include the creation time and assigner of a status
+- updating the getCurationStatus api call to return a JSON object for the status with label, assigner, and create time
+- adding an includeHistory query param for these API calls to allow seeing prior statuses
+- adding a facet to allow filtering by curation status (for users able to set them)
+- adding the creation time to solr as a pdate to support search by time period, e.g. current status set prior to a give date
+- standardizing the language around 'curation status' vs 'external status'
+- adding a 'curation-status' class to displayed labels to allow styling
+- adding a dataverse.ui.show-curation-status-to-all feature flag that allows users who can see a draft but not publish it to also view the curation status
+
+Due to changes in the solr schema, updating the solr schema and reindexing is required. Background reindexing should be OK.

doc/sphinx-guides/source/api/changelog.rst

@@ -11,6 +11,9 @@ v6.7
 ----
 
 - An undocumented :doc:`search` parameter called "show_my_data" has been removed. It was never exercised by tests and is believed to be unused. API users should use the :ref:`api-mydata` API instead.
+- /api/datasets/{id}/curationStatus API now includes a JSON object with curation label, createtime, and assigner rather than a string 'label' and it supports a new boolean includeHistory parameter (default false) that returns a JSON array of statuses
+- /api/datasets/{id}/listCurationStates includes new columns "Status Set Time" and "Status Set By" columns listing the time the current status was applied and by whom. It also supports the boolean includeHistory parameter. 
+- Due to updates in libraries used by Dataverse, XML serialization may have changed slightly with respect to whether self-closing tags are used for empty elements. This primiarily affects XML-based metadata exports. The XML structure of the export itself has not changed, so this is only an incompatibility if you are not using an XML parser.
 
 v6.6
 ----

doc/sphinx-guides/source/api/curation-labels.rst

@@ -1,16 +1,19 @@
-Dataset Curation Label API
-==========================
+Dataset Curation Status API
+===========================
 
 When the :ref:`:AllowedCurationLabels <:AllowedCurationLabels>` setting has been used to define Curation Labels, this API can be used to set these labels on draft datasets. 
 Superusers can define which set of labels are allowed for a given datasets in a collection/an individual dataset using the api described in the :doc:`/admin/dataverses-datasets` section.
 The API here can be used by curators/those who have permission to publish the dataset to get/set/change/delete the label currently assigned to a draft dataset.
+If the :ref:`dataverse.ui.show-curation-status-to-all` flag is enabled, users who can see the draft dataset version can use the get API call.
 
 This functionality is intended as a mechanism to integrate the Dataverse software with an external curation process/application: it is a way to make the state of a draft dataset, 
 as defined in the external process, visible within Dataverse. These labels have no other effect in Dataverse and are only visible to curators/those with permission to publish the dataset. 
 Any curation label assigned to a draft dataset will be removed upon publication.
+
+Dataverse tracks the Curation Label as well as when it was assigned and by whom. It also keeps track of the history of prior assignments.
 
-Get a Draft Dataset's Curation Label
-------------------------------------
+Get a Draft Dataset's Curation Status
+-------------------------------------
 
 .. code-block:: bash
 
@@ -27,8 +30,13 @@ Get a Draft Dataset's Curation Label
 
     curl -H X-Dataverse-key:$API_TOKEN "$SERVER_URL/api/datasets/:persistentId/curationStatus?persistentId=$DATASET_PID"
 
-You should expect a 200 ("OK") response and the draft dataset's curation status label contained in a JSON 'data' object.
+You should expect a 200 ("OK") response and the draft dataset's curation status as a JSON object contained in a JSON 'data' object. The status will include a 'label','createTime', and the 'assigner'.
+
+If the optional includeHistory query parameter is set to true, the responses 'data' entry will be a JSON array of curation status objects 
+
+    curl -H X-Dataverse-key:$API_TOKEN "$SERVER_URL/api/datasets/:persistentId/curationStatus?persistentId=$DATASET_PID&includeHistory=true"
 
+For draft datasets that were created prior to v6.7, it is possible that curation status objects will have no createTime or assigner.
 
 Set a Draft Dataset's Curation Label
 ------------------------------------
@@ -53,6 +61,8 @@ To add a curation label for a draft Dataset, specify the Dataset ID (DATASET_ID)
 
 You should expect a 200 ("OK") response indicating that the label has been set. 403/Forbidden and 400/Bad Request responses are also possible, i.e. if you don't have permission to make this change or are trying to add a label that isn't in the allowed set or to add a label to a dataset with no draft version.
 
+Note that Dataverse will add the current time as the createTime and the user as the 'assigner' of the label.
+
 
 Delete a Draft Dataset's Curation Label
 ---------------------------------------
@@ -98,7 +108,7 @@ You should expect a 200 ("OK") response with a comma-separated list of allowed l
 Get a Report on the Curation Status of All Datasets
 ---------------------------------------------------
 
-To get a CSV file listing the curation label assigned to each Dataset with a draft version, along with the creation and last modification dates, and list of those with permissions to publish the version.
+To get a CSV file listing the curation statuses assigned to each Dataset with a draft version, along with the creation and last modification dates, and list of those with permissions to publish the version.
 
 This API call is restricted to superusers.
 
@@ -112,3 +122,14 @@ This API call is restricted to superusers.
     curl -H X-Dataverse-key:$API_TOKEN "$SERVER_URL/api/datasets/listCurationStates"
 
 You should expect a 200 ("OK") response with a CSV formatted response.
+
+The CSV response includes the following columns in order:
+#. Dataset Title (as a hyperlink to the dataset page)
+#. Creation Date of the draft dataset version
+#. Latest Modification Date of the draft dataset version
+#. Assigned curation status or '<none>' if no curation status is assigned but was previously, null if no curation state has every been set.
+#. Time when the curation status was applied to the draft dataset version
+#. The user who assigned this curation status
+#. (and beyond): Users (comma separated list) with the Roles (column headings) that can publish datasets and therefore see/set curation status
+When includeHistory is true, multiple rows may be present for each dataset, showing the full history of curation statuses.
+

doc/sphinx-guides/source/installation/config.rst

@@ -1321,7 +1321,6 @@ List of S3 Storage Options
     dataverse.files.<id>.profile                 <?>                 Allows the use of AWS profiles for storage spanning multiple AWS accounts.           (none)
     dataverse.files.<id>.proxy-url               <?>                 URL of a proxy protecting the S3 store. Optional.                                    (none)
     dataverse.files.<id>.path-style-access       ``true``/``false``  Use path style buckets instead of subdomains. Optional.                              ``false``
-    dataverse.files.<id>.payload-signing         ``true``/``false``  Enable payload signing. Optional                                                     ``false``
     dataverse.files.<id>.chunked-encoding        ``true``/``false``  Disable chunked encoding. Optional                                                   ``true``
     dataverse.files.<id>.connection-pool-size    <?>                 The maximum number of open connections to the S3 server                              ``256``
     dataverse.files.<id>.disable-tagging         ``true``/``false``  Do not place the ``temp`` tag when redirecting the upload to the S3 server.          ``false``
@@ -1370,12 +1369,12 @@ Reported Working S3-Compatible Storage
   possibly slow) https://play.minio.io:9000 service.
 
 `StorJ Object Store <https://www.storj.io>`_
- StorJ is a distributed object store that can be configured with an S3 gateway. Per the S3 Storage instructions above, you'll first set up the StorJ S3 store by defining the id, type, and label. After following the general installation, set the following configurations to use a StorJ object store: ``dataverse.files.<id>.payload-signing=true`` and ``dataverse.files.<id>.chunked-encoding=false``. For step-by-step instructions see https://docs.storj.io/dcs/how-tos/dataverse-integration-guide/
+ StorJ is a distributed object store that can be configured with an S3 gateway. Per the S3 Storage instructions above, you'll first set up the StorJ S3 store by defining the id, type, and label. After following the general installation, set the following configuration to use a StorJ object store: ``dataverse.files.<id>.chunked-encoding=false``. For step-by-step instructions see https://docs.storj.io/dcs/how-tos/dataverse-integration-guide/
 
  Note that for direct uploads and downloads, Dataverse redirects to the proxy-url but presigns the urls based on the ``dataverse.files.<id>.custom-endpoint-url``. Also, note that if you choose to enable ``dataverse.files.<id>.download-redirect`` the S3 URLs expire after 60 minutes by default. You can change that minute value to reflect a timeout value that’s more appropriate by using ``dataverse.files.<id>.url-expiration-minutes``.
 
 `Surf Object Store v2019-10-30 <https://www.surf.nl/en>`_
-  Set ``dataverse.files.<id>.payload-signing=true``, ``dataverse.files.<id>.chunked-encoding=false`` and ``dataverse.files.<id>.path-style-request=true`` to use Surf Object
+  Set ``dataverse.files.<id>.chunked-encoding=false`` and ``dataverse.files.<id>.path-style-request=true`` to use Surf Object
   Store. You will need the Swift client (documented at <http://doc.swift.surfsara.nl/en/latest/Pages/Clients/s3cred.html>) to create the access key and secret key for the S3 interface.
 
 Note that the ``dataverse.files.<id>.proxy-url`` setting can be used in installations where the object store is proxied, but it should be considered an advanced option that will require significant expertise to properly configure. 
@@ -2265,7 +2264,7 @@ The S3 Archiver defines one custom setting, a required :S3ArchiverConfig. It can
 
 The credentials for your S3 account, can be stored in a profile in a standard credentials file (e.g. ~/.aws/credentials) referenced via "profile" key in the :S3ArchiverConfig setting (will default to the default entry), or can via MicroProfile settings as described for S3 stores (dataverse.s3archiver.access-key and dataverse.s3archiver.secret-key)
 
-The :S3ArchiverConfig setting is a JSON object that must include an "s3_bucket_name" and may include additional S3-related parameters as described for S3 Stores, including "profile", "connection-pool-size","custom-endpoint-url", "custom-endpoint-region", "path-style-access", "payload-signing", and "chunked-encoding".
+The :S3ArchiverConfig setting is a JSON object that must include an "s3_bucket_name" and may include additional S3-related parameters as described for S3 Stores, including "profile", "connection-pool-size","custom-endpoint-url", "custom-endpoint-region", "path-style-access", and "chunked-encoding".
 
 \:S3ArchiverConfig - minimally includes the name of the bucket to use. For example:
 
@@ -3270,6 +3269,21 @@ Defaults to ``true``.
 Can also be set via any `supported MicroProfile Config API source`_, e.g. the environment variable
 ``DATAVERSE_API_SHOW_LABEL_FOR_INCOMPLETE_WHEN_PUBLISHED``. Will accept ``[tT][rR][uU][eE]|1|[oO][nN]`` as "true" expressions.
 
+.. _dataverse.ui.show-curation-status-to-all:
+
+dataverse.ui.show-curation-status-to-all
+++++++++++++++++++++++++++++++++++++++++
+
+By default the curation status assigned to a draft dataset versioncan only be seen by those who can publish it. When this flag is true, anyone who can see the draft dataset can see the assigned status.
+These users will also get notifications/emails about changes to the status.
+See :ref:`:AllowedCurationLabels <:AllowedCurationLabels>` and the :doc:`/admin/dataverses-datasets` section for more information about curation status.
+
+Defaults to ``false``.
+
+Can also be set via any `supported MicroProfile Config API source`_, e.g. the environment variable
+``DATAVERSE_API_SHOW_CURATION_STATUS_TO_ALL``. Will accept ``[tT][rR][uU][eE]|1|[oO][nN]`` as "true" expressions.
+
+
 .. _dataverse.signposting.level1-author-limit:
 
 dataverse.signposting.level1-author-limit

docker-compose-dev.yml

@@ -209,7 +209,7 @@ services:
   dev_localstack:
     container_name: "dev_localstack"
     hostname: "localstack"
-    image: localstack/localstack:2.3.2
+    image: localstack/localstack:4.2.0
     restart: on-failure
     ports:
       - "127.0.0.1:4566:4566"

modules/dataverse-parent/pom.xml

@@ -32,12 +32,13 @@
                 <scope>import</scope>
             </dependency>
             <dependency>
-                <groupId>com.amazonaws</groupId>
-                <artifactId>aws-java-sdk-bom</artifactId>
+                <groupId>software.amazon.awssdk</groupId>
+                <artifactId>bom</artifactId>
                 <version>${aws.version}</version>
                 <type>pom</type>
                 <scope>import</scope>
             </dependency>
+
             <dependency>
                 <groupId>com.google.cloud</groupId>
                 <artifactId>libraries-bom</artifactId>
@@ -151,7 +152,7 @@
         <payara.version>6.2025.3</payara.version>
         <postgresql.version>42.7.7</postgresql.version>
         <solr.version>9.8.0</solr.version>
-        <aws.version>1.12.748</aws.version>
+        <aws.version>2.31.3</aws.version>
         <google.library.version>26.30.0</google.library.version>
 
         <!-- Basic libs, logging -->

pom.xml

@@ -77,6 +77,18 @@
                         <groupId>org.apache.geronimo.specs</groupId>
                         <artifactId>geronimo-javamail_1.4_spec</artifactId>
                     </exclusion>
+                    <exclusion>
+                        <groupId>org.apache.geronimo.specs</groupId>
+                        <artifactId>geronimo-stax-api_1.0_spec</artifactId>
+                    </exclusion>
+                    <exclusion>
+                        <groupId>org.codehaus.woodstox</groupId>
+                        <artifactId>wstx-asl</artifactId>
+                    </exclusion>
+                    <exclusion>
+                        <groupId>org.codehaus.woodstox</groupId>
+                        <artifactId>woodstox-core-asl</artifactId>
+                    </exclusion>
                 </exclusions>
             </dependency>
             <!-- Dependency for Apache Abdera and Apache Tika. Tika needs newer version. -->
@@ -167,10 +179,24 @@
                 </exclusion>
             </exclusions>
         </dependency>
-        
         <dependency>
-            <groupId>com.amazonaws</groupId>
-            <artifactId>aws-java-sdk-s3</artifactId>
+            <groupId>software.amazon.awssdk</groupId>
+            <artifactId>s3</artifactId>
+            <!-- no version here as managed by BOM above! -->
+        </dependency>
+        <dependency>
+            <groupId>software.amazon.awssdk</groupId>
+            <artifactId>s3-transfer-manager</artifactId>
+            <!-- no version here as managed by BOM above! -->
+        </dependency>
+        <!--dependency>
+            <groupId>software.amazon.awssdk</groupId>
+            <artifactId>apache-client</artifactId-->
+            <!-- no version here as managed by BOM above! -->
+        <!--/dependency-->
+        <dependency>
+            <groupId>software.amazon.awssdk</groupId>
+            <artifactId>netty-nio-client</artifactId>
             <!-- no version here as managed by BOM above! -->
         </dependency>
         <dependency>
@@ -181,7 +207,6 @@
         <dependency>
             <groupId>com.google.code.gson</groupId>
             <artifactId>gson</artifactId>
-            <version>2.9.1</version>
             <scope>compile</scope>
         </dependency>
         <!-- Should be refactored and moved to transitive section above once on Java EE 8 (makes WAR smaller) -->

scripts/zipdownload/pom.xml

@@ -23,8 +23,9 @@
             <artifactId>postgresql</artifactId>
         </dependency>
         <dependency>
-            <groupId>com.amazonaws</groupId>
-            <artifactId>aws-java-sdk-s3</artifactId>
+            <groupId>software.amazon.awssdk</groupId>
+            <artifactId>s3</artifactId>
+            <!-- no version here as managed by BOM above! -->
         </dependency>
     </dependencies>
     <build>