Merge branch 'develop' into mpconfig-personororg

# Conflicts:
#	src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java

Author: vera

conf/solr/solrconfig.xml

@@ -238,7 +238,7 @@
          have some sort of hard autoCommit to limit the log size.
       -->
     <autoCommit>
-      <maxTime>${solr.autoCommit.maxTime:30000}</maxTime>
+      <maxTime>${solr.autoCommit.maxTime:300000}</maxTime>
       <openSearcher>false</openSearcher>
     </autoCommit>
 

doc/release-notes/11374-indexing-improvement.md

@@ -0,0 +1,5 @@
+### Solr Indexing speed improved
+
+The performance of Solr indexing has been significantly improved, particularly for datasets with many files.
+
+A new dataverse.solr.min-files-to-use-proxy microprofile setting can be used to further improve performance/lower memory requirements for datasets with many files (e.g. 500+) (defaults to Integer.MAX, disabling use of the new functionality)

doc/release-notes/11414-add-dvobject-type-to-featured-items.md

@@ -0,0 +1,7 @@
+### Feature: Added dvObject and type fields to Featured Items 
+
+Dataverse Featured Items can now be linked to Dataverses, Datasets, or Datafiles.
+
+Pre-existing featured items as well as new items without dvObjects will be defaulted to type=custom.
+
+Featured Items with dvObjects will be filtered out of lists if the dvObject should not be viewed (i.e. datafiles that are restricted or datasets that are deaccessioned)

doc/release-notes/11438-fix-file-version-summary.md

@@ -0,0 +1,3 @@
+### File Version Summary Bug fix
+
+Fixes an issue where the deaccessionedReason was missing in the fileDifferenceSummary json object returned by API GET "$SERVER_URL/api/files/{ID}/versionDifferences"

doc/release-notes/11482-mpconfig-schema-dot-org-hide-download-urls.md

@@ -0,0 +1 @@
+The `dataverse.files.hide-schema-dot-org-download-urls` setting now supports configuration via MicroProfile Config. In addition to the existing `asadmin` JVM option method, any [supported MicroProfile Config API source](https://docs.payara.fish/community/docs/Technical%20Documentation/MicroProfile/Config/Overview.html) can now be used to set its value.

doc/release-notes/11489-CPUpdates.md

@@ -0,0 +1,5 @@
+Not sure what is already in the notes w.r.t. counter-processor updates (separate repo) so I'll try to be ~complete here
+
+### Make Data Count improvements
+
+Counter-processor, used to power Make Data Count metrics in Dataverse is now maintained in the https://github.com/gdcc/counter-processor repository. Multiple improvements to efficiency and scalability have been made. The example counter_daily.sh and counter_weekly.sh scripts that automate using counter-processor, available from the MDC section of the Dataverse Guides (https://guides.dataverse.org/en/latest/admin/make-data-count.html) have been updated to work with the latest counter-processor release and also have minor improvements.

doc/release-notes/11497-wholetale.md

@@ -0,0 +1 @@
+Unfortunately, the [Whole Tale](https://wholetale.org) project is no longer active and has been removed from the list of integrations in the Admin Guide.

doc/release-notes/Filter-efficiency.md

@@ -0,0 +1,70 @@
+### Improved efficiency for per-request Filters
+
+This release improves the performance of Dataverse's per-request handling of CORS Headers and API calls
+
+It adds new jvm-options/Microprofile settings replacing the now deprecated database settings.
+
+Additional changes:
+
+- CORS headers can now be configured with a list of desired origins, methods, and allowed and exposed headers.
+- An 'X-Dataverse-unblock-key' header has been added that can be used instead of the less secure 'unblock-key' query parameter when the :BlockedApiPolicy is set to 'unblock-key'
+- Warnings have been added to the log if the Blocked Api settings are misconfigured or if the key is weak (when the "unblock-key" policy is used)
+- The new `dataverse.api.blocked.key` can be configured using Payara password aliases or other secure storage options. 
+
+New JvmSettings:
+- `dataverse.cors.origin`: Allowed origins for CORS requests
+- `dataverse.cors.methods`: Allowed HTTP methods for CORS requests
+- `dataverse.cors.headers.allow`: Allowed headers for CORS requests
+- `dataverse.cors.headers.expose`: Headers to expose in CORS responses
+- `dataverse.api.blocked.policy`: Policy for blocking API endpoints
+- `dataverse.api.blocked.endpoints`: List of API endpoints to be blocked (comma-separated)
+- `dataverse.api.blocked.key`: Key for unblocking API endpoints
+
+Deprecated database settings:
+- `:AllowCors`
+- `:BlockedApiPolicy`
+- `:BlockedApiEndpoints`
+- `:BlockedApiKey`
+
+
+Upgrade instructions:
+
+The deprecated database settings will continue to work in this version. To use the new settings (which are more efficient),
+
+If :AllowCors is not set or is true:
+bin/asadmin create-jvm-options -Ddataverse.cors.origin=*
+
+Optionally set origin to a list of hosts and/or set other CORS JvmSettings
+Your currently blocked API endpoints can be found at http://localhost:8080/api/admin/settings/:BlockedApiEndpoints
+
+Copy them into the new setting with the following command. As with the deprecated setting, the endpoints should be comma-separated.
+
+bin/asadmin create-jvm-options '-Ddataverse.api.blocked.endpoints=<current :BlockedApiEndpoints>'
+
+If :BlockedApiPolicy is set and is not 'drop'
+bin/asadmin create-jvm-options '-Ddataverse.api.blocked.policy=<current :BlockedApiPolicy>'
+
+If :BlockedApiPolicy is 'unblock-key' and :BlockedApiKey is set
+
+   `echo "API_BLOCKED_KEY_ALIAS=<value of :BlockedApiKey>" > /tmp/dataverse.api.blocked.key.txt`
+
+   `sudo -u dataverse /usr/local/payara6/bin/asadmin create-password-alias --passwordfile /tmp/dataverse.api.blocked.key.txt`
+
+   When you are prompted "Enter the value for the aliasname operand", enter `api_blocked_key_alias`
+
+   You should see "Command create-password-alias executed successfully."
+
+   bin/asadmin create-jvm-options '-Ddataverse.api.blocked.key=${ALIAS=api_blocked_key_alias}'
+   
+   Restart Payara:
+   
+service payara restart
+
+Check server.log to verify that your new settings are in effect.
+   
+Cleanup: delete deprecated settings:
+curl -X DELETE http://localhost:8080/api/admin/settings/:AllowCors
+curl -X DELETE http://localhost:8080/api/admin/settings/:BlockedApiEndpoints
+curl -X DELETE http://localhost:8080/api/admin/settings/:BlockedApiPolicy
+curl -X DELETE http://localhost:8080/api/admin/settings/:BlockedApiKey
+

doc/sphinx-guides/source/_static/Dataverse_3ring-brand_icon_EqualSpace.svg

@@ -1,6 +1,5 @@
 Tool	Type	Scope	Description
 Data Explorer	explore	file	"A GUI which lists the variables in a tabular data file allowing searching, charting and cross tabulation analysis. The latest version incorporates the Data Curation Tool, a GUI for curating data by adding labels, groups, weights and other details to assist with informed reuse. See the README.md file at https://github.com/scholarsportal/Dataverse-Data-Explorer for the instructions on adding Data Explorer to your Dataverse."
-Whole Tale	explore	dataset	"A platform for the creation of reproducible research packages that allows users to launch containerized interactive analysis environments based on popular tools such as Jupyter and RStudio. Using this integration, Dataverse users can launch Jupyter and RStudio environments to analyze published datasets. For more information, see the `Whole Tale User Guide <https://wholetale.readthedocs.io/en/stable/users_guide/integration.html>`_."
 Binder	explore	dataset	Binder allows you to spin up custom computing environments in the cloud (including Jupyter notebooks) with the files from your dataset. See https://github.com/IQSS/dataverse-binder-redirect for installation instructions.
 File Previewers	explore	file	"A set of tools that display the content of files - including audio, html, `Hypothes.is <https://hypothes.is/>`_ annotations, images, PDF, Markdown, text, video, tabular data, spreadsheets, GeoJSON, zip, and NcML files - allowing them to be viewed without downloading the file. The previewers can be run directly from github.io, so the only required step is using the Dataverse API to register the ones you want to use. Documentation, including how to optionally brand the previewers, and an invitation to contribute through github are in the README.md file. Initial development was led by the Qualitative Data Repository and the spreasdheet previewer was added by the Social Sciences and Humanities Open Cloud (SSHOC) project. https://github.com/gdcc/dataverse-previewers"
 Ask the Data	query	file	Ask the Data is an experimental tool that allows you ask natural language questions about the data contained in Dataverse tables (tabular data). See the README.md file at https://github.com/IQSS/askdataverse/tree/main/askthedata for the instructions on adding Ask the Data to your Dataverse installation.