Merge branch 'develop' into 111-address-issues-with-file-search

Author: stevenwinship

doc/release-notes/11639-db-opts-idempotency.md

@@ -0,0 +1,45 @@
+## Database Settings Cleanup
+
+With this release, we remove some legacy specialties around Database Settings and provide better Admin API endpoints for them.
+
+Most important changes:
+
+1. Setting `BuiltinUsers.KEY` was renamed to `:BuiltinUsersKey`, aligned with our general naming pattern for options.
+2. Setting `WorkflowsAdmin#IP_WHITELIST_KEY` was renamed to `:WorkflowsAdminIpWhitelist`, aligned with our general naming pattern for options.
+3. Setting `:TabularIngestSizeLimit` no longer uses suffixes for formats and becomes a JSON-based setting instead.
+4. If set, all three settings will be migrated to their new form automatically for you (Flyway migration).
+5. You can no longer (accidentally) create or use arbitrary setting names or languages.
+   All Admin API endpoints for settings now validate setting names and languages for existence and compliance.
+
+As an administrator of a Dataverse instance, you can now make use of enhanced Bulk Operations on the Settings Admin API:
+
+1. Retrieving all settings as JSON via `GET /api/admin/settings` supports localized options now, too.
+2. You can replace all existing settings in an idempotent way sending JSON to `PUT /api/admin/settings`.
+   This will create, update and remove settings as necessary in one atomic operation.  
+   The new endpoint is especially useful to admins using GitOps or other automations.
+   It allows control over all Database Settings from a single source without risking an undefined state.
+
+Note: Despite the validation of setting names and languages, the content of any database setting is still not being validated when using the Settings Admin API!
+
+### Updated Database Settings
+
+The following database settings are were added to the official list within the code (to remain valid with the settings cleanup mentioned above):
+
+- `:BagGeneratorThreads`
+- `:BagItHandlerEnabled`
+- `:BagItLocalPath`
+- `:BagValidatorJobPoolSize`
+- `:BagValidatorJobWaitInterval`
+- `:BagValidatorMaxErrors`
+- `:BuiltinUsersKey` - formerly `BuiltinUsers.KEY`
+- `:CreateDataFilesMaxErrorsToDisplay`
+- `:DRSArchiverConfig` - a Harvard-specific setting
+- `:DuraCloudContext`
+- `:DuraCloudHost`
+- `:DuraCloudPort`
+- `:FileCategories`
+- `:GoogleCloudBucket`
+- `:GoogleCloudProject`
+- `:LDNAnnounceRequiredFields`
+- `:LDNTarget`
+- `:WorkflowsAdminIpWhitelist` - formerly `WorkflowsAdmin#IP_WHITELIST_KEY`

doc/release-notes/11744-cors-echo-origin-vary.md

@@ -0,0 +1,41 @@
+# 11744: CORS handling improvements
+
+Modernizes CORS so browser integrations (previewers, external tools, JS clients) work correctly with multiple origins and proper caching.
+
+## Highlights
+
+- Echoes the request origin (`Access-Control-Allow-Origin`) when it matches `dataverse.cors.origin`.
+- Adds `Vary: Origin` for per-origin responses (not for wildcard).
+- Supports comma‑separated origin list; any `*` in the list = wildcard mode.
+- CORS now only enabled when `dataverse.cors.origin` is set (removed `:AllowCors` no longer enables it).
+- All comma-separated configuration settings (database properties and MicroProfile config) now ignore spaces around commas; tokens remain unchanged (no quote parsing). Examples: `dataverse.cors.methods`, `dataverse.cors.headers.allow`, `dataverse.cors.headers.expose`. See "Comma-separated configuration values" in the Installation Guide.
+- Docs updated (Installation, Big Data Support, External Tools, File Previews); new tests cover edge cases.
+
+## Admin Action
+
+Set `dataverse.cors.origin` explicitly (required). Use explicit origins (not `*`) for credentialed requests. Ensure proxies keep `Vary: Origin`.
+
+Examples:
+
+```
+dataverse.cors.origin=https://example.org
+dataverse.cors.origin=https://libis.github.io,https://gdcc.github.io
+dataverse.cors.origin=*
+```
+
+Optional (unquoted):
+
+```
+dataverse.cors.methods=GET, POST, OPTIONS, PUT, DELETE
+```
+
+## Compatibility
+
+- Must configure `dataverse.cors.origin`; `:AllowCors` was deprecated and has now been removed.
+- Any `*` triggers wildcard (no per-origin echo / no Vary header).
+
+## Docs
+
+See updated `dataverse.cors.origin` section and related notes in Big Data Support (S3), External Tools, and File Previews.
+
+<!-- Maintainer note: The generic behavior for comma-separated settings has been documented centrally under Installation Guide > Configuration > "Comma-separated configuration values". Keep this item here as a cross-reference. -->

doc/release-notes/7618-file-level-permissions-restricted-draft.md

@@ -0,0 +1,3 @@
+File level permissions: Restricted files in Draft will now show a "Draft/Unpublished" tag in the UI when granting file access
+
+See #7618

doc/release-notes/metadataLanguage-API-call.md

@@ -0,0 +1,6 @@
+A new API endpoint has been implemented for getting the metadata language of a Dataverse Collection:
+
+`GET /dataverses/{alias}/allowedMetadataLanguages`: Returns the specified metadata language(s) in the collection if any.
+`PUT /dataverses/{alias}/allowedMetadataLanguages{metadataLanguage}`: Sets a metadata language in the collection.
+
+For more information, see #11856 and #11856.

doc/sphinx-guides/source/api/changelog.rst

@@ -9,7 +9,10 @@ This API changelog is experimental and we would love feedback on its usefulness.
 
 v6.9
 ----
+
 - The POST /api/admin/makeDataCount/{id}/updateCitationsForDataset processing is now asynchronous and the response no longer includes the number of citations. The response can be OK if the request is queued or 503 if the queue is full (default queue size is 1000).
+- The way to set per-format size limits for tabular ingest has changed. JSON input is now used. See :ref:`:TabularIngestSizeLimit`.
+- In the past, the settings API would accept any key and value. This is no longer the case because validation has been added. See :ref:`settings_put_single`, for example.
 
 v6.8
 ----

doc/sphinx-guides/source/api/external-tools.rst

@@ -11,6 +11,9 @@ Introduction
 
 External tools are additional applications the user can access or open from your Dataverse installation to preview, explore, and manipulate data files and datasets. The term "external" is used to indicate that the tool is not part of the main Dataverse Software.
 
+.. note::
+  Browser-based tools must have CORS explicitly enabled via :ref:`dataverse.cors.origin <dataverse.cors.origin>`. List every origin that will host your tool (or use ``*`` when a wildcard is acceptable). If an origin is not listed, the browser will block that tool's API requests even if the tool page itself loads.
+
 Once you have created the external tool itself (which is most of the work!), you need to teach a Dataverse installation how to construct URLs that your tool needs to operate. For example, if you've deployed your tool to fabulousfiletool.com your tool might want the ID of a file and the siteUrl of the Dataverse installation like this: https://fabulousfiletool.com?fileId=42&siteUrl=https://demo.dataverse.org
 
 In short, you will be creating a manifest in JSON format that describes not only how to construct URLs for your tool, but also what types of files your tool operates on, where it should appear in the Dataverse installation web interfaces, etc. 

doc/sphinx-guides/source/api/native-api.rst

@@ -287,6 +287,51 @@ The fully expanded example above (without environment variables) looks like this
 
   curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" "https://demo.dataverse.org/api/dataverses/root/roles"
 
+List the Allowed Metadata Languages of a Dataverse Collection
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Shows the allowed metadata languages of the Dataverse collection ``id``:
+
+.. code-block:: bash
+  
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export SERVER_URL=https://demo.dataverse.org
+  export ID=root
+
+  curl -H "X-Dataverse-key:$API_TOKEN" "$SERVER_URL/api/dataverses/$ID/allowedMetadataLanguages"
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" "https://demo.dataverse.org/api/dataverses/root/allowedMetadataLanguages"
+
+If there are no metadata languages configured on the server, this call returns an empty array. If the Dataverse collection has a mandatory metadata language, the return value is an array of that single language,
+otherwise it's an array of all available metadata languages on the server.
+
+Set the Allowed Metadata Language of a Dataverse Collection
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Sets the allowed metadata language of the Dataverse collection ``id`` to ``langCode`` if it's available on the server:
+
+.. code-block:: bash
+  
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export SERVER_URL=https://demo.dataverse.org
+  export ID=root
+  export LANGCODE=en
+
+  curl -H "X-Dataverse-key:$API_TOKEN" -X PUT "$SERVER_URL/api/dataverses/$ID/allowedMetadataLanguages/$LANGCODE"
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X PUT "https://demo.dataverse.org/api/dataverses/root/allowedMetadataLanguages/en"
+
+Returns an array of the set metadata language.
+If the metadata language is not available on the server, this call responds with a 400 BAD REQUEST.
+
 List Facets Configured for a Dataverse Collection
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -5843,13 +5888,13 @@ Builtin users are known as "Username/Email and Password" users in the :doc:`/use
 Create a Builtin User
 ~~~~~~~~~~~~~~~~~~~~~
 
-For security reasons, builtin users cannot be created via API unless the team who runs the Dataverse installation has populated a database setting called ``BuiltinUsers.KEY``, which is described under :ref:`securing-your-installation` and :ref:`database-settings` sections of Configuration in the Installation Guide. You will need to know the value of ``BuiltinUsers.KEY`` before you can proceed.
+For security reasons, builtin users cannot be created via API unless the team who runs the Dataverse installation has populated a database setting called ``:BuiltinUsersKey``, which is described under :ref:`securing-your-installation` and :ref:`database-settings` sections of Configuration in the Installation Guide. You will need to know the value of ``:BuiltinUsersKey`` before you can proceed.
 
 To create a builtin user via API, you must first construct a JSON document.  You can download :download:`user-add.json <../_static/api/user-add.json>` or copy the text below as a starting point and edit as necessary.
 
 .. literalinclude:: ../_static/api/user-add.json
 
-Place this ``user-add.json`` file in your current directory and run the following curl command, substituting variables as necessary. Note that both the password of the new user and the value of ``BuiltinUsers.KEY`` are passed as query parameters::
+Place this ``user-add.json`` file in your current directory and run the following curl command, substituting variables as necessary. Note that both the password of the new user and the value of ``:BuiltinUsersKey`` are passed as query parameters::
 
   curl -d @user-add.json -H "Content-type:application/json" "$SERVER_URL/api/builtin-users?password=$NEWUSER_PASSWORD&key=$BUILTIN_USERS_KEY"
 
@@ -7133,35 +7178,193 @@ If the PID is not managed by Dataverse, this call will report if the PID is reco
 Admin
 -----
 
-This is the administrative part of the API. For security reasons, it is absolutely essential that you block it before allowing public access to a Dataverse installation. Blocking can be done using settings. See the ``post-install-api-block.sh`` script in the ``scripts/api`` folder for details. See :ref:`blocking-api-endpoints` in Securing Your Installation section of the Configuration page of the Installation Guide.
+This is the administrative part of the API.
+For security reasons, it is absolutely essential that you block it before allowing public access to a Dataverse installation.
+See :ref:`blocking-api-endpoints` in the Installation Guide for details.
+
+.. note:: See :ref:`curl-examples-and-environment-variables` if you are unfamiliar with the use of export below.
+
+.. _admin-api-db-settings:
+
+Manage Database Settings
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+These are the API endpoints for managing the :ref:`database-settings` listed in the Installation Guide.
+
+.. _settings_get_all:
 
 List All Database Settings
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-List all settings::
+.. code-block:: bash
 
-  GET http://$SERVER/api/admin/settings
+  export SERVER_URL="http://localhost:8080"
+  
+  curl "$SERVER_URL/api/admin/settings"
 
-Configure Database Setting
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
 
-Sets setting ``name`` to the body of the request::
+  curl http://localhost:8080/api/admin/settings
 
-  PUT http://$SERVER/api/admin/settings/$name
+.. _settings_get_single:
 
 Get Single Database Setting
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Get the setting under ``name``::
+.. code-block:: bash
 
-  GET http://$SERVER/api/admin/settings/$name
+  export SERVER_URL="http://localhost:8080"
+  export NAME=":UploadMethods"
+  
+  curl "$SERVER_URL/api/admin/settings/$NAME"
 
-Delete Database Setting
-~~~~~~~~~~~~~~~~~~~~~~~
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl http://localhost:8080/api/admin/settings/:UploadMethods
+
+.. _settings_get_single_lang:
+
+Get Single Database Setting With Language/Locale
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A small number of settings, most notably :ref:`:ApplicationTermsOfUse`, can be saved in multiple languages.
+
+Use two-character ISO 639-1 language codes.
+
+.. code-block:: bash
+
+  export SERVER_URL="http://localhost:8080"
+  export NAME=":ApplicationTermsOfUse"
+  export LANG="en"
+  
+  curl "$SERVER_URL/api/admin/settings/$NAME/lang/$LANG"
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl http://localhost:8080/api/admin/settings/:ApplicationTermsOfUse/lang/en
 
-Delete the setting under ``name``::
+.. _settings_put_single:
+
+Configure Single Database Setting
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: bash
+
+  export SERVER_URL="http://localhost:8080"
+  export NAME=":InstallationName"
+  export VALUE="LibreScholar"
+  
+  curl -X PUT "$SERVER_URL/api/admin/settings/$NAME" -d "$VALUE"
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl -X PUT http://localhost:8080/api/admin/settings/:InstallationName -d LibreScholar
+
+Note: ``NAME`` values are validated for existence and compliance.
+
+.. _settings_put_single_lang:
+
+Configure Single Database Setting With Language/Locale
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A small number of settings, most notably :ref:`:ApplicationTermsOfUse`, can be saved in multiple languages.
+
+Use two-character ISO 639-1 language codes.
+
+.. code-block:: bash
+
+  export SERVER_URL="http://localhost:8080"
+  export NAME=":ApplicationTermsOfUse"
+  export LANG="fr"
+  
+  curl -X PUT "$SERVER_URL/api/admin/settings/$NAME/lang/$LANG" --upload-file /tmp/apptou_fr.html
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl -X PUT http://localhost:8080/api/admin/settings/:ApplicationTermsOfUse/lang/fr --upload-file /tmp/apptou_fr.html
+
+Note: ``NAME`` and ``LANG`` values are validated for existence and compliance.
+
+.. _settings_put_bulk:
+
+Configure All Database Settings
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Using a JSON file, replace all settings in a single idempotent and atomic operation and delete any settings not present in that JSON file.
+
+Use the JSON ``data`` object in output of ``GET /api/admin/settings`` (:ref:`settings_get_all`) for the JSON input structure for this endpoint.
+To put this concretely, you can save just the ``data`` object for your existing settings to disk by filtering them through ``jq`` like this:
+
+.. code-block:: bash
+
+  curl http://localhost:8080/api/admin/settings | jq '.data' > /tmp/all-settings.json
+
+Then you can use this "all-settings.json" file as a starting point for your input file.
+The :doc:`../installation/config` page of the Installation Guide has a :ref:`complete list of all the available settings <database-settings>`.
+Note that settings in the JSON file are validated for existence and compliance.
+
+.. code-block:: bash
+
+  export SERVER_URL="http://localhost:8080"
+  
+  curl -X PUT -H "Content-type:application/json" "$SERVER_URL/api/admin/settings" --upload-file /tmp/all-settings.json
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl -X PUT -H "Content-type:application/json" http://localhost:8080/api/admin/settings --upload-file /tmp/all-settings.json
+
+.. _settings_delete_single:
+
+Delete Single Database Setting
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: bash
+
+  export SERVER_URL="http://localhost:8080"
+  export NAME=":InstallationName"
+  
+  curl -X DELETE "$SERVER_URL/api/admin/settings/$NAME"
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl -X DELETE http://localhost:8080/api/admin/settings/:InstallationName
+
+.. _settings_delete_single_lang:
+
+Delete Single Database Setting With Language/Locale
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A small number of settings, most notably :ref:`:ApplicationTermsOfUse`, can be saved in multiple languages.
+
+Use two-character ISO 639-1 language codes.
+
+.. code-block:: bash
+
+  export SERVER_URL="http://localhost:8080"
+  export NAME=":ApplicationTermsOfUse"
+  export LANG="fr"
+  
+  curl -X DELETE "$SERVER_URL/api/admin/settings/$NAME/lang/$LANG"
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
 
-  DELETE http://$SERVER/api/admin/settings/$name
+  curl -X DELETE http://localhost:8080/api/admin/settings/:ApplicationTermsOfUse/lang/fr 
 
 .. _list-all-feature-flags: