Merge branch 'develop' into 10340-forbidden #10340

Author: pdurbin

.github/workflows/spi_release.yml

@@ -45,7 +45,7 @@ jobs:
                   server-id: ossrh
                   server-username: MAVEN_USERNAME
                   server-password: MAVEN_PASSWORD
-            - uses: actions/cache@v2
+            - uses: actions/cache@v4
               with:
                   path: ~/.m2
                   key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }}
@@ -68,7 +68,7 @@ jobs:
                 with:
                     java-version: '17'
                     distribution: 'adopt'
-            -   uses: actions/cache@v2
+            -   uses: actions/cache@v4
                 with:
                     path: ~/.m2
                     key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }}

conf/solr/schema.xml

@@ -0,0 +1,4 @@
+Extension of API `{id}/versions` and `{id}/versions/{versionId}` with an optional ``excludeMetadataBlocks`` parameter,
+that specifies whether the metadataBlocks should be listed in the output. It defaults to ``false``, preserving backward
+compatibility. (Note that for a dataset with a large number of versions and/or metadataBlocks having the metadata blocks
+included can dramatically increase the volume of the output). See also [the guides](https://dataverse-guide--10778.org.readthedocs.build/en/10778/api/native-api.html#list-versions-of-a-dataset), #10778, and #10171.

doc/release-notes/10171-exlude-metadatablocks.md

@@ -0,0 +1,3 @@
+### Broken Link in Email When Users Request Access to Files
+
+When users request access to a files, the people who have permission to grant access receive an email with a link in it that didn't work due to a trailing period (full stop) right next to the link (e.g. `https://demo.dataverse.org/permissions-manage-files.xhtml?id=9.`) A space has been added to fix this. See #10384 and #11115.

doc/release-notes/10384-link.md

@@ -0,0 +1,82 @@
+This release enhances how numerical and date fields are indexed in Solr. Previously, all fields were indexed as English text (text_en), but with this update:
+
+* Integer fields are indexed as `plong`
+* Float fields are indexed as `pdouble`
+* Date fields are indexed as `date_range` (`solr.DateRangeField`)
+
+Specifically, the following fields were updated:
+
+- coverage.Depth
+- coverage.ObjectCount
+- coverage.ObjectDensity
+- coverage.Redshift.MaximumValue
+- coverage.Redshift.MinimumValue
+- coverage.RedshiftValue
+- coverage.SkyFraction
+- coverage.Spectral.CentralWavelength
+- coverage.Spectral.MaximumWavelength
+- coverage.Spectral.MinimumWavelength
+- coverage.Temporal.StartTime
+- coverage.Temporal.StopTime
+- dateOfCollectionEnd
+- dateOfCollectionStart
+- dateOfDeposit
+- distributionDate
+- dsDescriptionDate
+- journalPubDate
+- productionDate
+- resolution.Redshift
+- targetSampleActualSize
+- timePeriodCoveredEnd
+- timePeriodCoveredStart
+
+This change enables range queries when searching from both the UI and the API, such as `dateOfDeposit:[2000-01-01 TO 2014-12-31]` or `targetSampleActualSize:[25 TO 50]`.
+
+Dataverse administrators must update their Solr schema.xml (manually or by rerunning `update-fields.sh`) and reindex all datasets.
+
+Additionally, search result highlighting is now more accurate, ensuring that only fields relevant to the query are highlighted in search results. If the query is specifically limited to certain fields, the highlighting is now limited to those fields as well.
+
+## Upgrade Instructions
+
+7\. Update Solr schema.xml file. Start with the standard v6.5 schema.xml, then, if your installation uses any custom or experimental metadata blocks, update it to include the extra fields (step 7a).
+
+Stop Solr (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.5/installation/prerequisites.html#solr-init-script)).
+
+```shell
+service solr stop
+```
+
+Replace schema.xml
+
+```shell
+wget https://raw.githubusercontent.com/IQSS/dataverse/v6.5/conf/solr/schema.xml
+cp schema.xml /usr/local/solr/solr-9.4.1/server/solr/collection1/conf
+```
+
+Start Solr (but if you use any custom metadata blocks, perform the next step, 7a first).
+
+```shell
+service solr start
+```
+
+7a\. For installations with custom or experimental metadata blocks:
+
+Before starting Solr, update the schema to include all the extra metadata fields that your installation uses. We do this by collecting the output of the Dataverse schema API and feeding it to the `update-fields.sh` script that we supply, as in the example below (modify the command lines as needed to reflect the names of the directories, if different):
+
+```shell
+	wget https://raw.githubusercontent.com/IQSS/dataverse/v6.5/conf/solr/update-fields.sh
+	chmod +x update-fields.sh
+	curl "http://localhost:8080/api/admin/index/solr/schema" | ./update-fields.sh /usr/local/solr/solr-9.4.1/server/solr/collection1/conf/schema.xml
+```
+
+Now start Solr.
+
+8\. Reindex Solr
+
+Below is the simplest way to reindex Solr:
+
+```shell
+curl http://localhost:8080/api/admin/index
+```
+
+The API above rebuilds the existing index "in place". If you want to be absolutely sure that your index is up-to-date and consistent, you may consider wiping it clean and reindexing everything from scratch (see [the guides](https://guides.dataverse.org/en/latest/admin/solr-search-index.html)). Just note that, depending on the size of your database, a full reindex may take a while and the users will be seeing incomplete search results during that window.

doc/release-notes/10887-solr-field-types.md

@@ -0,0 +1,14 @@
+## New APIs for External Tools Registration for Marketplace
+
+New API base path /api/externalTools created that mimics the admin APIs /api/admin/externalTools. These new add and delete apis require an authenticated superuser token.
+
+Example:
+```
+   API_TOKEN='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
+   export TOOL_ID=1
+
+   curl http://localhost:8080/api/externalTools
+   curl http://localhost:8080/api/externalTools/$TOOL_ID
+   curl -s -H "X-Dataverse-key:$API_TOKEN" -X POST -H 'Content-type: application/json' http://localhost:8080/api/externalTools --upload-file fabulousFileTool.json
+   curl -s -H "X-Dataverse-key:$API_TOKEN" -X DELETE http://localhost:8080/api/externalTools/$TOOL_ID
+```

doc/release-notes/10930-marketplace-external-tools-apis.md

@@ -0,0 +1,11 @@
+## Feature: Prevent publishing Datasets without files
+A new attribute was added to Collections in order to control the publishing of Datasets without files. 
+Once set to "True", the publishing of a Dataset within a Collection, without files, will be blocked for all non superusers.
+In order to configure a Collection to block publishing a superuser must set the attribute "requireFilesToPublishDataset" to "True".
+The collection's hierarchy will be checked if the collection's "requireFilesToPublishDataset" attribute is not set explicitly to "True" or "False".
+
+```shell
+curl -X PUT -H "X-Dataverse-key:$API_TOKEN" "$SERVER_URL/api/dataverses/$ID/attribute/requireFilesToPublishDataset?value=true"
+```
+
+See also #10981.

doc/release-notes/10981-published-datasets-should-contain-files.md

@@ -0,0 +1 @@
+Metadata Customization guide has been updated to explain how to implement a kind of boolean fieldtype (see [Metadata Customization Guide](https://guides.dataverse.org/en/latest/admin/metadatacustomization.html#controlledvocabulary-enumerated-properties))

doc/release-notes/11064-update-metadata-customization.md

@@ -0,0 +1,6 @@
+## Improvement and internationalization of harvest status
+
+Added a harvest status to differentiate a complete harvest with errors (Completed with failures) and without errors (Completed)
+Harvest status labels are now internationalized 
+
+For more information, see issue [#9294](https://github.com/IQSS/dataverse/issues/9294)

doc/release-notes/9294-improvement-and-internationalization-of-harvest-status.md

@@ -35,7 +35,13 @@ Configure the tool with the curl command below, making sure to replace the ``fab
 
 .. code-block:: bash
 
-  curl -X POST -H 'Content-type: application/json' http://localhost:8080/api/admin/externalTools --upload-file fabulousFileTool.json 
+  curl -X POST -H 'Content-type: application/json' http://localhost:8080/api/admin/externalTools --upload-file fabulousFileTool.json
+
+This API is Superuser only. Note the endpoint difference (/api/externalTools instead of /api/admin/externalTools).
+
+.. code-block:: bash
+
+  curl -s -H "X-Dataverse-key:$API_TOKEN" -X POST -H 'Content-type: application/json' http://localhost:8080/api/externalTools --upload-file fabulousFileTool.json
 
 Listing All External Tools in a Dataverse Installation
 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
@@ -46,6 +52,12 @@ To list all the external tools that are available in a Dataverse installation:
 
   curl http://localhost:8080/api/admin/externalTools
 
+This API is open to any user. Note the endpoint difference (/api/externalTools instead of /api/admin/externalTools).
+
+.. code-block:: bash
+
+  curl http://localhost:8080/api/externalTools
+
 Showing an External Tool in a Dataverse Installation
 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 
@@ -56,6 +68,12 @@ To show one of the external tools that are available in a Dataverse installation
   export TOOL_ID=1
   curl http://localhost:8080/api/admin/externalTools/$TOOL_ID
 
+This API is open to any user. Note the endpoint difference (/api/externalTools instead of /api/admin/externalTools).
+
+.. code-block:: bash
+
+  curl http://localhost:8080/api/externalTools/$TOOL_ID
+
 Removing an External Tool From a Dataverse Installation
 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
@@ -66,6 +84,12 @@ Assuming the external tool database id is "1", remove it with the following comm
   export TOOL_ID=1
   curl -X DELETE http://localhost:8080/api/admin/externalTools/$TOOL_ID
 
+This API is Superuser only. Note the endpoint difference (/api/externalTools instead of /api/admin/externalTools).
+
+.. code-block:: bash
+
+  curl -s -H "X-Dataverse-key:$API_TOKEN" -X DELETE http://localhost:8080/api/externalTools/$TOOL_ID
+
 .. _testing-external-tools:
 
 Testing External Tools