Merge remote-tracking branch 'IQSS/develop' into ORCID_1

Author: qqmyers

doc/release-notes/11163-CSL citation formats.md

@@ -0,0 +1,5 @@
+This release adds support for generating citations in any of the standard independent formats specified using the [Citation Style Language](https://citationstyles.org/).
+The CSL formats are available to copy/paste from a new "Cite Dataset" menu "View Styles Citations" pop-up the dataset page.
+An API call to retrieve a dataset citation in EndNote, RIS, BibTeX, and CSLJson format has also been added. (The first three have been available as downloads from the UI but have not been directly accessible via API. The CSLJson format is new to Dataverse and can be used with open-source libraries to generate all of the other CSL stypes citations.)
+
+Admins can use a new dataverse.csl.common-styles setting to highlight commonly used styles - common styles are listed in the pop-up, others can be found by type-ahead search in a list of 1000+ options.

doc/release-notes/9302-add-Etalab-License.md

@@ -2,4 +2,4 @@ A new file has been added to import the French Open License to Dataverse: licens
 
 This license, which is compatible with the Creative Commons license, is recommended by the French government for open documents.
 
-For more information, see #9301 and #9302.
+For more information, see #9301, #9302, and #11302.

doc/sphinx-guides/source/_static/admin/counter-processor-config.yaml

@@ -33,8 +33,10 @@ path_types:
 
 # Robots and machines urls are urls where the script can download a list of regular expressions to determine
 # if something is a robot or machine user-agent.  The text file has one regular expression per line
-robots_url: https://raw.githubusercontent.com/CDLUC3/Make-Data-Count/master/user-agents/lists/robot.txt
-machines_url: https://raw.githubusercontent.com/CDLUC3/Make-Data-Count/master/user-agents/lists/machine.txt
+#robots_url: https://raw.githubusercontent.com/CDLUC3/Make-Data-Count/master/user-agents/lists/robot.txt
+#machines_url: https://raw.githubusercontent.com/CDLUC3/Make-Data-Count/master/user-agents/lists/machine.txt
+robots_url: https://raw.githubusercontent.com/IQSS/counter-processor/refs/heads/goto-gdcc/user-agents/lists/robots.txt
+machines_url: https://raw.githubusercontent.com/IQSS/counter-processor/refs/heads/goto-gdcc/user-agents/lists/machine.txt
 
 # the year and month for the report you are creating.
 year_month: 2019-01

doc/sphinx-guides/source/_static/util/counter_daily.sh

@@ -1,6 +1,6 @@
 #! /bin/bash
 
-COUNTER_PROCESSOR_DIRECTORY="/usr/local/counter-processor-1.05"
+COUNTER_PROCESSOR_DIRECTORY="/usr/local/counter-processor-1.06"
 MDC_LOG_DIRECTORY="/usr/local/payara6/glassfish/domains/domain1/logs/mdc"
 
 # counter_daily.sh

doc/sphinx-guides/source/admin/make-data-count.rst

@@ -84,9 +84,9 @@ Configure Counter Processor
 
 * Change to the directory where you installed Counter Processor.
 
-  * ``cd /usr/local/counter-processor-1.05``
+  * ``cd /usr/local/counter-processor-1.06``
 
-* Download :download:`counter-processor-config.yaml <../_static/admin/counter-processor-config.yaml>` to ``/usr/local/counter-processor-1.05``.
+* Download :download:`counter-processor-config.yaml <../_static/admin/counter-processor-config.yaml>` to ``/usr/local/counter-processor-1.06``.
 
 * Edit the config file and pay particular attention to the FIXME lines.
 
@@ -99,7 +99,7 @@ Soon we will be setting up a cron job to run nightly but we start with a single
 
 * Change to the directory where you installed Counter Processor.
 
-  * ``cd /usr/local/counter-processor-1.05``
+  * ``cd /usr/local/counter-processor-1.06``
 
 * If you are running Counter Processor for the first time in the middle of a month, you will need create blank log files for the previous days. e.g.:
 

doc/sphinx-guides/source/api/native-api.rst

@@ -3050,7 +3050,7 @@ The fully expanded example above (without environment variables) looks like this
 .. code-block:: bash
 
   curl "https://demo.dataverse.org/api/datasets/:persistentId/makeDataCount/citations?persistentId=10.5072/FK2/J8SJZB"
-
+  
 Delete Unpublished Dataset
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -3317,6 +3317,8 @@ Usage example:
 Get Citation
 ~~~~~~~~~~~~
 
+This API call returns the dataset citation as seen on the dataset page, wrapped as a JSON object, with the value in the "data" sub-object's "message" key.
+
 .. code-block:: bash
 
   export SERVER_URL=https://demo.dataverse.org
@@ -3334,6 +3336,35 @@ Usage example:
 .. code-block:: bash
 
   curl -H "Accept:application/json" "$SERVER_URL/api/datasets/:persistentId/versions/$VERSION/{version}/citation?persistentId=$PERSISTENT_IDENTIFIER&includeDeaccessioned=true"
+  
+Get Citation In Other Formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Dataverse can also generate dataset citations in "EndNote", "RIS", "BibTeX", and "CSLJson" formats.
+Unlike the call above, which wraps the result in JSON, this API call sends the raw format with the appropriate content-type (EndNote is XML, RIS and BibTeX are plain text, and CSLJson is JSON). ("Internal" is also a valid value, returning the same content as the above call as HTML).
+This API call adds a format parameter in the API call which can be any of the values listed above.
+
+Usage example:
+
+.. code-block:: bash
+
+  export SERVER_URL=https://demo.dataverse.org
+  export PERSISTENT_IDENTIFIER=doi:10.5072/FK2/YD5QDG
+  export VERSION=1.0
+  export FORMAT=EndNote
+
+  curl "$SERVER_URL/api/datasets/:persistentId/versions/$VERSION/{version}/citation/$FORMAT?persistentId=$PERSISTENT_IDENTIFIER"
+
+By default, deaccessioned dataset versions are not included in the search when applying the :latest or :latest-published identifiers. Additionally, when filtering by a specific version tag, you will get a "not found" error if the version is deaccessioned and you do not enable the ``includeDeaccessioned`` option described below.
+
+If you want to include deaccessioned dataset versions, you must set ``includeDeaccessioned`` query parameter to ``true``.
+
+Usage example:
+
+.. code-block:: bash
+
+  curl "$SERVER_URL/api/datasets/:persistentId/versions/$VERSION/{version}/citation/$FORMAT?persistentId=$PERSISTENT_IDENTIFIER&includeDeaccessioned=true"
+  
 
 Get Citation by Preview URL Token
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

doc/sphinx-guides/source/developers/make-data-count.rst

@@ -49,7 +49,7 @@ Once you are done with your configuration, you can run Counter Processor like th
 
 ``su - counter``
 
-``cd /usr/local/counter-processor-1.05``
+``cd /usr/local/counter-processor-1.06``
 
 ``CONFIG_FILE=counter-processor-config.yaml python39 main.py``
 
@@ -82,7 +82,7 @@ Second, if you are also sending your SUSHI report to Make Data Count, you will n
 
 ``curl -H "Authorization: Bearer $JSON_WEB_TOKEN" -X DELETE https://$MDC_SERVER/reports/$REPORT_ID``
 
-To get the ``REPORT_ID``, look at the logs generated in ``/usr/local/counter-processor-1.05/tmp/datacite_response_body.txt``
+To get the ``REPORT_ID``, look at the logs generated in ``/usr/local/counter-processor-1.06/tmp/datacite_response_body.txt``
 
 To read more about the Make Data Count api, see https://github.com/datacite/sashimi
 
@@ -110,9 +110,11 @@ The script will process the newest set of log files (merging files from multiple
 APIs to manage the states include GET, POST, and DELETE (for testing), as shown below.
 
 Note: ``yearMonth`` must be in the format ``yyyymm`` or ``yyyymmdd``.
+Note: If running the new script on multiple servers add the query parameter &server=serverName on the first POST call. The server name can not be changed once set. To clear the name out you must delete the state and post a new one.
 
 ``curl -X GET http://localhost:8080/api/admin/makeDataCount/{yearMonth}/processingState``
 
+``curl -X POST http://localhost:8080/api/admin/makeDataCount/{yearMonth}/processingState?state=processing&server=server1``
 ``curl -X POST http://localhost:8080/api/admin/makeDataCount/{yearMonth}/processingState?state=done``
 
 ``curl -X DELETE http://localhost:8080/api/admin/makeDataCount/{yearMonth}/processingState``

doc/sphinx-guides/source/installation/config.rst

@@ -3446,6 +3446,16 @@ dataverse.files.globus-monitoring-server
 
 This setting is required in conjunction with the ``globus-use-experimental-async-framework`` feature flag (see :ref:`feature-flags`). Setting it to true designates the Dataverse instance to serve as the dedicated polling server. It is needed so that the new framework can be used in a multi-node installation. 
 
+.. _dataverse.csl.common-styles:
+
+dataverse.csl.common-styles
++++++++++++++++++++++++++++
+
+This setting allows admins to highlight a few of the 1000+ CSL citation styles available from the dataset page. The value should be a comma-separated list of styles.
+These will be listed above the alphabetical list of all styles in the "View Styled Citations" pop-up.
+The default value when not set is "chicago-author-date, ieee". 
+
+
 .. _feature-flags:
 
 Feature Flags

doc/sphinx-guides/source/installation/prerequisites.rst

@@ -438,9 +438,9 @@ A scripted installation using Ansible is mentioned in the :doc:`/developers/make
 As root, download and install Counter Processor::
 
         cd /usr/local
-        wget https://github.com/gdcc/counter-processor/archive/refs/tags/v1.05.tar.gz
-        tar xvfz v1.05.tar.gz
-        cd /usr/local/counter-processor-1.05
+        wget https://github.com/gdcc/counter-processor/archive/refs/tags/v1.06.tar.gz
+        tar xvfz v1.06.tar.gz
+        cd /usr/local/counter-processor-1.06
 
 Installing GeoLite Country Database
 ===================================
@@ -451,7 +451,7 @@ The process required to sign up, download the database, and to configure automat
 
 As root, change to the Counter Processor directory you just created, download the GeoLite2-Country tarball from MaxMind, untar it, and copy the geoip database into place::
 
-        <download or move the GeoLite2-Country.tar.gz to the /usr/local/counter-processor-1.05 directory>
+        <download or move the GeoLite2-Country.tar.gz to the /usr/local/counter-processor-1.06 directory>
         tar xvfz GeoLite2-Country.tar.gz
         cp GeoLite2-Country_*/GeoLite2-Country.mmdb maxmind_geoip
 
@@ -461,12 +461,12 @@ Creating a counter User
 As root, create a "counter" user and change ownership of Counter Processor directory to this new user::
 
         useradd counter
-        chown -R counter:counter /usr/local/counter-processor-1.05
+        chown -R counter:counter /usr/local/counter-processor-1.06
 
 Installing Counter Processor Python Requirements
 ================================================
 
-Counter Processor version 1.05 requires Python 3.7 or higher. This version of Python is available in many operating systems, and is purportedly available for RHEL7 or CentOS 7 via Red Hat Software Collections. Alternately, one may compile it from source.
+Counter Processor version 1.06 requires Python 3.7 or higher.
 
 The following commands are intended to be run as root but we are aware that Pythonistas might prefer fancy virtualenv or similar setups. Pull requests are welcome to improve these steps!
 
@@ -477,7 +477,7 @@ Install Python 3.9::
 Install Counter Processor Python requirements::
 
         python3.9 -m ensurepip
-        cd /usr/local/counter-processor-1.05
+        cd /usr/local/counter-processor-1.06
         pip3 install -r requirements.txt
 
 See the :doc:`/admin/make-data-count` section of the Admin Guide for how to configure and run Counter Processor.

doc/sphinx-guides/source/user/dataset-management.rst

@@ -173,7 +173,7 @@ Certain file types in the Dataverse installation are supported by additional fun
 File Previews
 -------------
 
-Dataverse installations can add previewers for common file types uploaded by their research communities. The previews appear on the file page. If a preview tool for a specific file type is available, the preview will be created and will display automatically, after terms have been agreed to or a guestbook entry has been made, if necessary. File previews are not available for restricted files unless they are being accessed using a Preview URL. See also :ref:`previewUrl`.
+Dataverse installations can add previewers for common file types uploaded by their research communities. The previews appear on the file page. If a preview tool for a specific file type is available, the preview will be created and will display automatically, after terms have been agreed to or a guestbook entry has been made, if necessary. File previews are not available for restricted files unless they are being accessed using a Preview URL. See also :ref:`previewUrl`. When the dataset license is not the default license, users will be prompted to accept the license/data use agreement before the preview is shown. See also :ref:`license-terms`.
 
 Previewers are available for the following file types: