Merge branch 'develop' into 9301-etalab_license_enhancements #9302 #11302

Author: pdurbin

conf/solr/schema.xml

@@ -206,7 +206,8 @@
     <field name="deaccessionReason" type="string" stored="true" indexed="false" multiValued="false"/>
     <field name="fileRestricted" type="boolean" stored="true" indexed="false" multiValued="false"/>
     <field name="canDownloadFile" type="boolean" stored="true" indexed="false" multiValued="false"/>
-
+    <field name="versionNote" type="string" stored="true" indexed="false" multiValued="false"/>
+    
     <!-- Added for Dataverse 4.0 alpha 1. This is a required field so we don't have to go to the database to get the database id of the entity. On cards we use the id in links -->
     <field name="entityId" type="plong" stored="true" indexed="true" multiValued="false"/>
 

doc/release-notes/10476-display-on-create-field-option.md

@@ -0,0 +1,10 @@
+### OAI Harvesting from DataCite
+
+DataCite maintains an OAI server (https://oai.datacite.org/oai) that serves records for every DOI they have registered. There's been a lot of interest in the community in being able to harvest from them. This way, it will be possible to harvest metadata from institution X even if the institution X does not maintain an OAI server of their own, if they happen to register their DOIs with DataCite. One extra element of this harvesting model that makes it especially powerful and flexible is the DataCite's concept of a "dynamic OAI set": a harvester is not limited to harvesting the pre-defined set of ALL the records registered by the Institution X, but can instead harvest virtually any arbitrary subset thereof; any query that the DataCite search API understands can be used as an OAI set (!). The feature is already in use at IQSS, as a beta version patch. 
+
+For various reasons, in order to take advantage of this feature harvesting clients must be created using the `/api/harvest/clients` API. Once configured however, harvests can be run from the Harvesting Clients control panel in the UI.
+
+DataCite-harvesting clients must be configured with 2 new feature flags, `useListRecords` and `useOaiIdentifiersAsPids` (added in v6.5). Note that these features may be of use when harvesting from other sources, not just from DataCite.
+
+See "Harvesting from DataCite" under https://guides.dataverse.org/en/latest/api/native-api.html#managing-harvesting-clients for more information.
+

doc/release-notes/10909-datacite-oai-harvesting.md

@@ -0,0 +1,5 @@
+This release adds support for generating citations in any of the standard independent formats specified using the [Citation Style Language](https://citationstyles.org/).
+The CSL formats are available to copy/paste from a new "Cite Dataset" menu "View Styles Citations" pop-up the dataset page.
+An API call to retrieve a dataset citation in EndNote, RIS, BibTeX, and CSLJson format has also been added. (The first three have been available as downloads from the UI but have not been directly accessible via API. The CSLJson format is new to Dataverse and can be used with open-source libraries to generate all of the other CSL stypes citations.)
+
+Admins can use a new dataverse.csl.common-styles setting to highlight commonly used styles - common styles are listed in the pop-up, others can be found by type-ahead search in a list of 1000+ options.

doc/release-notes/11163-CSL citation formats.md

@@ -0,0 +1,13 @@
+### Metadata Source Facet Can Now Differentiate Between Harvested Sources
+
+The behavior of the feature flag `index-harvested-metadata-source` and the "Metadata Source" facet, which were added and updated, respectively, in [Dataverse 6.3](https://github.com/IQSS/dataverse/releases/tag/v6.3) (through pull requests #10464 and #10651), have been updated. A new field called "Source Name" has been added to harvesting clients.
+
+Before Dataverse 6.3, all harvested content (datasets and files) appeared together under "Harvested" under the "Metadata Source" facet. This is still the behavior of Dataverse out of the box. Since Dataverse 6.3, enabling the `index-harvested-metadata-source` feature flag (and reindexing) resulted in harvested content appearing under the nickname for whatever harvesting client was used to bring in the content. This meant that instead of having all harvested content lumped together under "Harvested", content would appear under "client1", "client2", etc.
+
+Now, as this release, enabling the `index-harvested-metadata-source` feature flag, populating a new field for harvesting clients called "Source Name" ("sourceName" in the [API](https://dataverse-guide--11217.org.readthedocs.build/en/11217/api/native-api.html#create-a-harvesting-client)), and reindexing (see upgrade instructions below), results in the source name appearing under the "Metadata Source" facet rather than the harvesting client nickname. This gives you more control over the name that appears under the "Metadata Source" facet and allows you to group harvested content from various harvesting clients under the same name if you wish (by reusing the same source name).
+
+Previously, `index-harvested-metadata-source` was not documented in the guides, but now you can find information about it under [Feature Flags](https://dataverse-guide--11217.org.readthedocs.build/en/11217/installation/config.html#feature-flags). See also #10217 and #11217.
+
+## Upgrade instructions
+
+If you have enabled the `dataverse.feature.index-harvested-metadata-source` feature flag and given some of your harvesting clients a source name, you should reindex to have those source names appear under the "Metadata Source" facet.

doc/release-notes/11217-source-name-harvesting-client.md

@@ -0,0 +1,3 @@
+A bug that caused ORCIDs start with https://orcid.org/ entered as author identifier to be ignored when creating the DataCite metadata has been fixed. This primarily affected users of the ORCID external vocabulary script as the manual entry form recommends not using the URL form.
+
+The display of authorIdentifier, when not using any external vocabulary scripts, has been improved so that either the plain identifier (e.g. "0000-0002-1825-0097") or its URL form (e.g. "https://orcid.org/0000-0002-1825-0097") will result in valid links in the display (for identifier types that have a URL form). The URL form is now recommended when doing manual entry.

doc/release-notes/11242-fix-oricid-recognition.md

@@ -0,0 +1,12 @@
+### New API for the SPA, in order to replicate the "classic" download count.
+- ``includeMDC`` parameter is optional. False or not included will return the total count of downloads if MDC is not running. Or, if MDCStartDate is set the count returned will be limited to the time prior to the MDCStartDate. Setting ``includeMDC`` to true will ignore any MDCStartDate and return the total download count for the dataset.
+
+Example:
+```
+   API_TOKEN='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
+   export DATASET_ID=1
+   export includeMDC=False
+
+   curl -s -H "X-Dataverse-key:$API_TOKEN" -X GET http://localhost:8080/api/datasets/$DATASET_ID/download/count
+   curl -s -H "X-Dataverse-key:$API_TOKEN" -X GET http://localhost:8080/api/datasets/$DATASET_ID/download/count?includeMDC=true
+```

doc/release-notes/11244-get-dataset-download-count-api.md

@@ -0,0 +1,12 @@
+Dataverse now supports the option of adding a versionNote before/during publication of a dataset that can be used to indicate why a version was created and/or how it differs from the prior version. Whether this feature is enabled is controlled by a flag. Version notes are shown in the user interface (dataset page version table), indexed, available via the API, and have been added to the JSON, DDI, DataCite, and OAI-ORE exports.
+
+With the addition of this feature, work has been done to clean-up and rename fields that have been used for specifying the reason for deaccessioning a dataset and providing an optional link to a non-Dataverse location where the dataset still can be found. The former was listed in some JSON-based API calls and exports as "versionNote" and is now "deaccessionNote", while the latter was referred to as "archiveNote" and is now "deacccessionLink". These result in incompatibilities in the UI related to deaccessioned datasets.
+
+Further, some database consolidation has been done to combine the deaccessionlink and archivenote fields which appear to have both been used for the same purpose (the deaccessionlink db field is older and was not displayed in the current UI. Going forward, only the deaccessionlink column exists.
+
+New Feature Flags:
+
+VERSION_NOTE - false by default.
+
+Update of the solr schema (using the standard instructions) is needed.
+

doc/release-notes/8431-versionNotes.md

@@ -0,0 +1,11 @@
+{
+  "nickName": "zenodo",
+  "dataverseAlias": "zenodoHarvested",
+  "harvestUrl": "https://zenodo.org/oai2d",
+  "archiveUrl": "https://zenodo.org",
+  "archiveDescription": "Moissonné depuis la collection LMOPS de l'entrepôt Zenodo. En cliquant sur ce jeu de données, vous serez redirigé vers Zenodo.",
+  "metadataFormat": "oai_dc",
+  "customHeaders": "x-oai-api-key: xxxyyyzzz",
+  "set": "user-lmops",
+  "allowHarvestingMissingCVV":true
+}

doc/sphinx-guides/source/_static/api/harvesting-client.json

@@ -25,6 +25,12 @@ Please note that in some rare cases this GUI may fail to create a client because
 
 Note that as of 5.13, a new entry "Custom HTTP Header" has been added to the Step 1. of Create or Edit form. This optional field can be used to configure this client with a specific HTTP header to be added to every OAI request. This is to accommodate a (rare) use case where the remote server may require a special token of some kind in order to offer some content not available to other clients. Most OAI servers offer the same publicly-available content to all clients, so few admins will have a use for this feature. It is however on the very first, Step 1. screen in case the OAI server requires this token even for the "ListSets" and "ListMetadataFormats" requests, which need to be sent in the Step 2. of creating or editing a client. Multiple headers can be supplied separated by `\\n` - actual "backslash" and "n" characters, not a single "new line" character. 
 
+Harvesting from Datacite
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+As of v6.6, it is now possible to harvest metadata directly from DataCite. Their OAI gateway (https://oai.datacite.org/oai) serves records for every DOI they have registered. Therefore, it is now possible to harvest metadata from any participating institution even if they do not maintain an OAI server of their own. Their OAI implementation offers a concept of a "dynamic set", making it possible to use any query supported by the DataCite search API as though it were a "set". This makes harvesting from them extra flexible, allowing to harvest virtually any arbitrary subset of metadata records, potentially spanning multiple institutions and registration authorities.
+
+For various reasons, in order to take advantage of this feature harvesting clients must be created via the ``/api/harvest/clients`` API. Once configured however, harvests can be run from the Harvesting Clients control panel in the UI. See the :ref:`managing-harvesting-clients-api` section of the :doc:`/api/native-api` guide for more information.
 
 How to Stop a Harvesting Run in Progress
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -56,4 +62,4 @@ Harvesting Client Changelog
 Harvesting Non-OAI-PMH
 ~~~~~~~~~~~~~~~~~~~~~~
 
-`DOI2PMH <https://github.com/IQSS/doi2pmh-server>`__ is a community-driven project intended to allow OAI-PMH harvesting from non-OAI-PMH sources.  
+`DOI2PMH <https://github.com/IQSS/doi2pmh-server>`__ is a community-driven project intended to allow OAI-PMH harvesting from non-OAI-PMH sources.