Merge branch 'develop' into 360-modify-notvalid-deaccession-reason

Author: stevenwinship

doc/release-notes/11804-notifications-api-updates.md

@@ -0,0 +1,21 @@
+## Notifications API Update
+
+**Endpoint:** `notifications/all`
+
+**Enhancements:**
+
+* When the query parameter `inAppNotificationFormat=true` is set:
+
+    * Notifications of types:
+
+        * `REQUESTFILEACCESS`
+        * `REQUESTEDFILEACCESS`
+        * `GRANTFILEACCESS`
+        * `REJECTFILEACCESS`
+
+  now return both the **dataset display name** and **dataset persistent identifier**.
+
+    * Notifications of type `DATASETMENTIONED` now return a **formatted JSON** in the `additionalInfo` field when this field contains a valid persisted JSON string, instead of a raw JSON string.
+
+Related issue: #11804
+Related PR: #11851

doc/release-notes/11828-unacceptable-performance-deleting-datasets.md

@@ -0,0 +1 @@
+This release adds database indexes on GuestbookResponse and DatasetMetrtics, speeding up Dataset deletes. It also adds a constraint preventing null VersionState, as a matter of good housekeeping practice. 

doc/release-notes/11850-big-data-admin-guide.md

@@ -0,0 +1,3 @@
+### Big Data Admin Section
+
+- A new section - Scaling Dataverse with Data Size - has been added to the Admin Guide. It is intended to help administrators configure Dataverse appropriately to handle larger amounts of data.

doc/release-notes/11852-notifications-api-pagination-and-only-unread.md

@@ -0,0 +1,72 @@
+## Notifications API Update
+
+**Endpoint:** `notifications/all`
+
+The user notifications endpoint has been enhanced with new optional query parameters to allow for more specific and
+efficient data retrieval.
+
+**1. Filter by Unread Status**
+
+You can now fetch only unread notifications by using the `onlyUnread` boolean parameter.
+
+* **`onlyUnread`**: (Optional, boolean) When set to `true`, the API will only return notifications that the user has not
+  yet marked as read.
+
+**Example:**
+
+```bash
+curl -H "X-Dataverse-key:$API_TOKEN" "$SERVER_URL/api/notifications/all?onlyUnread=true"
+```
+
+**2. Pagination Support**
+
+Pagination is now supported through the limit and offset parameters, allowing you to retrieve notifications in smaller,
+manageable chunks.
+
+- **`limit`**: (Optional, integer) Specifies the maximum number of notifications to return.
+
+- **`offset`**: (Optional, integer) Specifies the number of notifications to skip before starting to return results.
+
+Example (Retrieve notifications 11 through 20):
+
+```bash
+curl -H "X-Dataverse-key:$API_TOKEN" "$SERVER_URL/api/notifications/all?limit=10&offset=10"
+```
+
+**3 Breaking Change: API Response Format Updated**
+
+To support pagination and improve consistency across the API, the JSON response format for this endpoint has been
+changed. This is a breaking change and will require updates to any client code currently using this endpoint.
+
+**Old Format:**
+
+Previously, the response nested the notification list inside a notifications object within the data field.
+
+```
+{
+  "status": "OK",
+  "data": {
+    "notifications": [
+      / ... /
+    ]
+  }
+}
+```
+
+**New Format:**
+
+The response now includes a top-level totalCount field (required for pagination) and places the notification list
+directly in the data field. This flattens the structure and makes it consistent with other paginated endpoints.
+
+```
+{
+  "status": "OK",
+  "totalCount": 2,
+  "data": [
+    / ... /  
+  ]
+}
+```
+
+Related issue: #11852
+Related PR: #11854

doc/sphinx-guides/source/admin/big-data-administration.rst

@@ -35,3 +35,4 @@ This guide documents the functionality only available to superusers (such as "da
    maintenance
    backups
    troubleshooting
+   big-data-administration

doc/sphinx-guides/source/admin/index.rst

@@ -13,13 +13,14 @@ v6.9
 - The POST /api/admin/makeDataCount/{id}/updateCitationsForDataset processing is now asynchronous and the response no longer includes the number of citations. The response can be OK if the request is queued or 503 if the queue is full (default queue size is 1000).
 - The way to set per-format size limits for tabular ingest has changed. JSON input is now used. See :ref:`:TabularIngestSizeLimit`.
 - In the past, the settings API would accept any key and value. This is no longer the case because validation has been added. See :ref:`settings_put_single`, for example.
+- For GET /api/notifications/all the JSON response has changed breaking the backward compatibility of the API.
 
 v6.8
 ----
 
 - For POST /api/files/{id}/metadata passing an empty string ("description":"")  or array ("categories":[]) will no longer be ignored. Empty fields will now clear out the values in the file's metadata. To ignore the fields simply do not include them in the JSON string.
 - For PUT /api/datasets/{id}/editMetadata the query parameter "sourceInternalVersionNumber" has been removed and replaced with "sourceLastUpdateTime" to verify that the data being edited hasn't been modified and isn't stale.
-- For GET /api/dataverses/$dataverse-alias/links the Json response has changed breaking the backward compatibility of the API.
+- For GET /api/dataverses/$dataverse-alias/links the JSON response has changed breaking the backward compatibility of the API.
 - For GET /api/admin/dataverse/{dataverse-alias}/storageDriver and /api/datasets/{identifier}/storageDriver the driver name is no longer returned in data.message. This value is now returned in data.name.
 - For PUT /api/dataverses/$dataverse-alias/inputLevels custom input levels that had been previously set will no longer be deleted. To delete input levels send an empty list (deletes all), then send the new/modified list.
 - For GET /api/externalTools and /api/externalTools/{id} the responses are now formatted as JSON (previously the toolParameters and allowedApiCalls were a JSON object and array (respectively) that were serialized as JSON strings) and any configured "requirements" are included.

doc/sphinx-guides/source/api/changelog.rst

@@ -6520,44 +6520,65 @@ The expected OK (200) response looks something like this:
 
   {
       "status": "OK",
-      "data": {
-          "notifications": [
-              {
-                  "id": 38,
-                  "type": "CREATEACC",
-                  "displayAsRead": true,
-                  "subjectText": "Root: Your account has been created",
-                  "messageText": "Hello, \nWelcome to...",
-                  "sentTimestamp": "2025-07-21T19:15:37Z"
-              }
+      "totalCount": 15,
+      "data": [
+        {
+            "id": 38,
+            "type": "CREATEACC",
+            "displayAsRead": true,
+            "subjectText": "Root: Your account has been created",
+            "messageText": "Hello, \nWelcome to...",
+            "sentTimestamp": "2025-07-21T19:15:37Z"
+        }
   ...
 
-This endpoint supports an optional query parameter ``inAppNotificationFormat`` which, if sent as ``true``, retrieves the fields needed to build the in-app notifications for the Notifications section of the Dataverse UI, omitting fields related to email notifications.
+This endpoint supports several optional query parameters to filter and paginate the results.
+
+The ``inAppNotificationFormat`` parameter, if sent as ``true``, retrieves the fields needed to build the in-app notifications for the Notifications section of the Dataverse UI, omitting fields related to email notifications.
 
 .. code-block:: bash
 
   curl -H "X-Dataverse-key:$API_TOKEN" "$SERVER_URL/api/notifications/all?inAppNotificationFormat=true"
 
-The expected OK (200) response looks something like this:
+The ``onlyUnread`` parameter, if sent as ``true``, filters the results to include only notifications that have not been marked as read.
+
+.. code-block:: bash
+
+  curl -H "X-Dataverse-key:$API_TOKEN" "$SERVER_URL/api/notifications/all?onlyUnread=true"
+
+The ``limit`` and ``offset`` parameters can be used for pagination. ``limit`` specifies the maximum number of notifications to return, and ``offset`` specifies the number of notifications to skip from the beginning of the list. For example, to retrieve notifications 11 through 15:
+
+To aid in pagination the JSON response also includes the total number of rows (totalCount) available.
+
+.. code-block:: bash
+
+  curl -H "X-Dataverse-key:$API_TOKEN" "$SERVER_URL/api/notifications/all?limit=5&offset=10"
+
+All parameters can be combined. For instance, to get the first page of 10 unread notifications in the in-app format:
+
+.. code-block:: bash
+
+  curl -H "X-Dataverse-key:$API_TOKEN" "$SERVER_URL/api/notifications/all?inAppNotificationFormat=true&onlyUnread=true&limit=1&offset=0"
+
+The expected OK (200) response for an in-app format request looks something like this:
 
 .. code-block:: text
 
   {
       "status": "OK",
-      "data": {
-          "notifications": [
-              {
-                  "id": 79,
-                  "type": "CREATEACC",
-                  "displayAsRead": false,
-                  "sentTimestamp": "2025-08-08T08:00:16Z",
-                  "installationBrandName": "Your Installation Name",
-                  "userGuidesBaseUrl": "https://guides.dataverse.org",
-                  "userGuidesVersion": "6.7.1",
-                  "userGuidesSectionPath": "user/index.html"
-              }
-          ]
-      }
+      "totalCount": 15,
+      "data": [
+        {
+            "id": 79,
+            "type": "CREATEACC",
+            "displayAsRead": false,
+            "sentTimestamp": "2025-08-08T08:00:16Z",
+            "installationBrandName": "Your Installation Name",
+            "userGuidesBaseUrl": "https://guides.dataverse.org",
+            "userGuidesVersion": "6.7.1",
+            "userGuidesSectionPath": "user/index.html"
+        }
+      ]
   }
   ...
 

doc/sphinx-guides/source/api/native-api.rst

@@ -1,6 +1,14 @@
 Production
 ==========
 
+.. _production-security-warning:
+
+.. warning::
+
+    The :doc:`demo` tutorial is **NOT SECURE BY DEFAULT**. It uses public, hardcoded passwords and secrets for demonstration purposes only.
+
+    If you use the demo as a structural template, you MUST replace all default secrets before deploying your instance. Failure to do so will result in a vulnerable production environment.
+
 .. contents:: |toctitle|
 	:local:
 
@@ -11,7 +19,7 @@ As of Dataverse 6.8, when we introduced image tagging per version (see the :ref:
 
 The images and the documentation is not perfect, of course.
 
-For now, we recommend following the :doc:`demo` tutorial. It will help you learn how to configure and secure your installation. Not that instead of "latest" you might want to select a specific version. Again see :ref:`app-image-supported-tags`.
+For now, we recommend following the :doc:`demo` as a structural template. Note that instead of "latest" you might want to select a specific version. Again see :ref:`app-image-supported-tags`.
 
 The Dataverse guides were originally written with a non-Docker installation in mind so we'd like rewrite them with both Docker and non-Docker in mind. This is a big job, obviously. 😅 We know we'd like to write more about ports. We'd like to explain `how to set up Rserve <https://github.com/IQSS/dataverse/issues/11731>`_. Etc., etc.
 
@@ -30,4 +38,4 @@ Please try the images (see :doc:`demo`) and give feedback (see :ref:`helping-con
 Alternatives
 ------------
 
-The traditional (non-Docker) installation method is described in the :doc:`/installation/index`.
+The traditional (non-Docker) installation method is described in the :doc:`/installation/index`.

doc/sphinx-guides/source/container/running/production.rst

@@ -196,6 +196,6 @@ As described in that document, Globus transfers can be initiated by choosing the
 
 An overview of the control and data transfer interactions between components was presented at the 2022 Dataverse Community Meeting and can be viewed in the `Integrations and Tools Session Video <https://youtu.be/3ek7F_Dxcjk?t=5289>`_ around the 1 hr 28 min mark.
 
-See also :ref:`Globus settings <:GlobusSettings>`.
+See also :ref:`Globus settings <:GlobusSettings>` and :ref:`globus-stores`.
 
 An alternative, experimental implementation of Globus polling of ongoing upload transfers has been added in v6.4. This framework does not rely on the instance staying up continuously for the duration of the transfer and saves the state information about Globus upload requests in the database. Due to its experimental nature it is not enabled by default. See the ``globus-use-experimental-async-framework`` feature flag (see :ref:`feature-flags`) and the JVM option :ref:`dataverse.files.globus-monitoring-server`.