Merge branch 'develop' into 11959-automate-solr-schema-updates

Author: poikilotherm

.github/workflows/deploy_beta_testing.yml

@@ -69,7 +69,7 @@ jobs:
           overwrite: true
 
       - name: Execute payara war deployment remotely
-        uses: appleboy/ssh-action@v1.2.2
+        uses: appleboy/ssh-action@v1.2.3
         env:
           INPUT_WAR_FILE: ${{ env.war_file }}
         with:

doc/release-notes/11639-db-opts-idempotency.md

@@ -0,0 +1,45 @@
+## Database Settings Cleanup
+
+With this release, we remove some legacy specialties around Database Settings and provide better Admin API endpoints for them.
+
+Most important changes:
+
+1. Setting `BuiltinUsers.KEY` was renamed to `:BuiltinUsersKey`, aligned with our general naming pattern for options.
+2. Setting `WorkflowsAdmin#IP_WHITELIST_KEY` was renamed to `:WorkflowsAdminIpWhitelist`, aligned with our general naming pattern for options.
+3. Setting `:TabularIngestSizeLimit` no longer uses suffixes for formats and becomes a JSON-based setting instead.
+4. If set, all three settings will be migrated to their new form automatically for you (Flyway migration).
+5. You can no longer (accidentally) create or use arbitrary setting names or languages.
+   All Admin API endpoints for settings now validate setting names and languages for existence and compliance.
+
+As an administrator of a Dataverse instance, you can now make use of enhanced Bulk Operations on the Settings Admin API:
+
+1. Retrieving all settings as JSON via `GET /api/admin/settings` supports localized options now, too.
+2. You can replace all existing settings in an idempotent way sending JSON to `PUT /api/admin/settings`.
+   This will create, update and remove settings as necessary in one atomic operation.  
+   The new endpoint is especially useful to admins using GitOps or other automations.
+   It allows control over all Database Settings from a single source without risking an undefined state.
+
+Note: Despite the validation of setting names and languages, the content of any database setting is still not being validated when using the Settings Admin API!
+
+### Updated Database Settings
+
+The following database settings are were added to the official list within the code (to remain valid with the settings cleanup mentioned above):
+
+- `:BagGeneratorThreads`
+- `:BagItHandlerEnabled`
+- `:BagItLocalPath`
+- `:BagValidatorJobPoolSize`
+- `:BagValidatorJobWaitInterval`
+- `:BagValidatorMaxErrors`
+- `:BuiltinUsersKey` - formerly `BuiltinUsers.KEY`
+- `:CreateDataFilesMaxErrorsToDisplay`
+- `:DRSArchiverConfig` - a Harvard-specific setting
+- `:DuraCloudContext`
+- `:DuraCloudHost`
+- `:DuraCloudPort`
+- `:FileCategories`
+- `:GoogleCloudBucket`
+- `:GoogleCloudProject`
+- `:LDNAnnounceRequiredFields`
+- `:LDNTarget`
+- `:WorkflowsAdminIpWhitelist` - formerly `WorkflowsAdmin#IP_WHITELIST_KEY`

doc/release-notes/11744-cors-echo-origin-vary.md

@@ -0,0 +1,41 @@
+# 11744: CORS handling improvements
+
+Modernizes CORS so browser integrations (previewers, external tools, JS clients) work correctly with multiple origins and proper caching.
+
+## Highlights
+
+- Echoes the request origin (`Access-Control-Allow-Origin`) when it matches `dataverse.cors.origin`.
+- Adds `Vary: Origin` for per-origin responses (not for wildcard).
+- Supports comma‑separated origin list; any `*` in the list = wildcard mode.
+- CORS now only enabled when `dataverse.cors.origin` is set (removed `:AllowCors` no longer enables it).
+- All comma-separated configuration settings (database properties and MicroProfile config) now ignore spaces around commas; tokens remain unchanged (no quote parsing). Examples: `dataverse.cors.methods`, `dataverse.cors.headers.allow`, `dataverse.cors.headers.expose`. See "Comma-separated configuration values" in the Installation Guide.
+- Docs updated (Installation, Big Data Support, External Tools, File Previews); new tests cover edge cases.
+
+## Admin Action
+
+Set `dataverse.cors.origin` explicitly (required). Use explicit origins (not `*`) for credentialed requests. Ensure proxies keep `Vary: Origin`.
+
+Examples:
+
+```
+dataverse.cors.origin=https://example.org
+dataverse.cors.origin=https://libis.github.io,https://gdcc.github.io
+dataverse.cors.origin=*
+```
+
+Optional (unquoted):
+
+```
+dataverse.cors.methods=GET, POST, OPTIONS, PUT, DELETE
+```
+
+## Compatibility
+
+- Must configure `dataverse.cors.origin`; `:AllowCors` was deprecated and has now been removed.
+- Any `*` triggers wildcard (no per-origin echo / no Vary header).
+
+## Docs
+
+See updated `dataverse.cors.origin` section and related notes in Big Data Support (S3), External Tools, and File Previews.
+
+<!-- Maintainer note: The generic behavior for comma-separated settings has been documented centrally under Installation Guide > Configuration > "Comma-separated configuration values". Keep this item here as a cross-reference. -->

doc/release-notes/11832-DataCite-scaling.md

@@ -0,0 +1,14 @@
+This release adds functionality to retry calls to DataCite when their server is overloaded or Dataverse has hit their rate limit.
+
+It also introduces an option to only update DataCite metadata after checking to see if the current DataCite information is out of date.
+(This adds a request to get information from DataCite before any potential write of new information which will be more efficient when 
+most DOIs have not changed but will result in an extra call to get info when a DOI has changed.)
+  
+Both of these can help when DataCite is being used heavily, e.g. creating and publishing datasets with many datafiles and using file DOIs,
+or doing bulk operations that involve DataCite with many datasets.
+
+### New Settings
+
+- dataverse.feature.only-update-datacite-when-needed 
+
+The default is false - Dataverse will not check to see if DataCite's information is out of date before sending an update.

doc/release-notes/11921-dataset-version-summaries-metadatacount-fix.md

@@ -0,0 +1,10 @@
+### Dataset version summaries API changedFileMetaData count fix
+
+The endpoint ``{id}/versions/compareSummary`` was previously returning an incorrect count for
+the ``changedFileMetaData`` field.
+The logic for calculating this count has been fixed to accurately reflect the total number of file metadata changes
+across all files in the dataset version.
+
+### Related issues
+
+- https://github.com/IQSS/dataverse/issues/11921

doc/release-notes/7618-file-level-permissions-restricted-draft.md

@@ -0,0 +1,3 @@
+File level permissions: Restricted files in Draft will now show a "Draft/Unpublished" tag in the UI when granting file access
+
+See #7618

doc/release-notes/metadataLanguage-API-call.md

@@ -0,0 +1,6 @@
+A new API endpoint has been implemented for getting the metadata language of a Dataverse Collection:
+
+`GET /dataverses/{alias}/allowedMetadataLanguages`: Returns the specified metadata language(s) in the collection if any.
+`PUT /dataverses/{alias}/allowedMetadataLanguages{metadataLanguage}`: Sets a metadata language in the collection.
+
+For more information, see #11856 and #11856.

doc/sphinx-guides/source/api/changelog.rst

@@ -9,7 +9,10 @@ This API changelog is experimental and we would love feedback on its usefulness.
 
 v6.9
 ----
+
 - The POST /api/admin/makeDataCount/{id}/updateCitationsForDataset processing is now asynchronous and the response no longer includes the number of citations. The response can be OK if the request is queued or 503 if the queue is full (default queue size is 1000).
+- The way to set per-format size limits for tabular ingest has changed. JSON input is now used. See :ref:`:TabularIngestSizeLimit`.
+- In the past, the settings API would accept any key and value. This is no longer the case because validation has been added. See :ref:`settings_put_single`, for example.
 
 v6.8
 ----

doc/sphinx-guides/source/api/external-tools.rst

@@ -11,6 +11,9 @@ Introduction
 
 External tools are additional applications the user can access or open from your Dataverse installation to preview, explore, and manipulate data files and datasets. The term "external" is used to indicate that the tool is not part of the main Dataverse Software.
 
+.. note::
+  Browser-based tools must have CORS explicitly enabled via :ref:`dataverse.cors.origin <dataverse.cors.origin>`. List every origin that will host your tool (or use ``*`` when a wildcard is acceptable). If an origin is not listed, the browser will block that tool's API requests even if the tool page itself loads.
+
 Once you have created the external tool itself (which is most of the work!), you need to teach a Dataverse installation how to construct URLs that your tool needs to operate. For example, if you've deployed your tool to fabulousfiletool.com your tool might want the ID of a file and the siteUrl of the Dataverse installation like this: https://fabulousfiletool.com?fileId=42&siteUrl=https://demo.dataverse.org
 
 In short, you will be creating a manifest in JSON format that describes not only how to construct URLs for your tool, but also what types of files your tool operates on, where it should appear in the Dataverse installation web interfaces, etc.