Merge branch 'develop' into 11639-db-opts-idempotency

sekmiller · sekmiller · commit 757f2e54b17b · 2025-11-18T09:45:09.000-05:00
diff --git a/.github/workflows/deploy_beta_testing.yml b/.github/workflows/deploy_beta_testing.yml
@@ -69,7 +69,7 @@ jobs:
           overwrite: true
 
       - name: Execute payara war deployment remotely
-        uses: appleboy/ssh-action@v1.2.2
+        uses: appleboy/ssh-action@v1.2.3
         env:
           INPUT_WAR_FILE: ${{ env.war_file }}
         with:
diff --git a/doc/release-notes/11832-DataCite-scaling.md b/doc/release-notes/11832-DataCite-scaling.md
@@ -0,0 +1,14 @@
+This release adds functionality to retry calls to DataCite when their server is overloaded or Dataverse has hit their rate limit.
+
+It also introduces an option to only update DataCite metadata after checking to see if the current DataCite information is out of date.
+(This adds a request to get information from DataCite before any potential write of new information which will be more efficient when 
+most DOIs have not changed but will result in an extra call to get info when a DOI has changed.)
+  
+Both of these can help when DataCite is being used heavily, e.g. creating and publishing datasets with many datafiles and using file DOIs,
+or doing bulk operations that involve DataCite with many datasets.
+
+### New Settings
+
+- dataverse.feature.only-update-datacite-when-needed 
+
+The default is false - Dataverse will not check to see if DataCite's information is out of date before sending an update.
diff --git a/doc/sphinx-guides/source/container/app-image.rst b/doc/sphinx-guides/source/container/app-image.rst
@@ -17,11 +17,6 @@ Within the main repository, you may find the application image's files at ``<git
 This is the same Maven module providing a Dataverse WAR file for classic installations, and uses the
 `Maven Docker Plugin <https://dmp.fabric8.io>`_ to build and ship the image within a special Maven profile.
 
-**NOTE: This image is created, maintained and supported by the Dataverse community on a best-effort basis.**
-IQSS will not offer you support how to deploy or run it, please reach out to the community for help on using it.
-You might be interested in taking a look at :doc:`../developers/containers`, linking you to some (community-based)
-efforts.
-
 .. _app-image-supported-tags:
 
 Supported Image Tags
diff --git a/doc/sphinx-guides/source/container/base-image.rst b/doc/sphinx-guides/source/container/base-image.rst
@@ -16,11 +16,6 @@ Within the main repository, you may find the base image's files at ``<git root>/
 This Maven module uses the `Maven Docker Plugin <https://dmp.fabric8.io>`_ to build and ship the image.
 You may use, extend, or alter this image to your liking and/or host in some different registry if you want to.
 
-**NOTE: This image is created, maintained and supported by the Dataverse community on a best-effort basis.**
-IQSS will not offer you support how to deploy or run it, please reach out to the community (:ref:`support`) for help on using it.
-You might be interested in taking a look at :doc:`../developers/containers`, linking you to some (community-based)
-efforts.
-
 .. _base-image-supported-tags:
 
 Supported Image Tags
diff --git a/doc/sphinx-guides/source/container/dev-usage.rst b/doc/sphinx-guides/source/container/dev-usage.rst
@@ -1,7 +1,7 @@
 Development Usage
 =================
 
-Please note! This Docker setup is not for production!
+Please note! This Docker setup is not for :doc:`production <running/production>`!
 
 .. contents:: |toctitle|
         :local:
diff --git a/doc/sphinx-guides/source/container/intro.rst b/doc/sphinx-guides/source/container/intro.rst
@@ -9,7 +9,7 @@ Dataverse in containers!
 Intended Audience
 -----------------
 
-This guide is intended for anyone who wants to run Dataverse in containers. This is potentially a wide audience, from sysadmins interested in running Dataverse in production in containers (not recommended yet) to contributors working on a bug fix (encouraged!). See :doc:`running/index` for various scenarios and please let us know if your use case is not covered.
+This guide is intended for anyone who wants to run Dataverse in containers. This is potentially a wide audience, from sysadmins interested in running Dataverse in production in containers to contributors working on a bug fix. See :doc:`running/index` for various scenarios and please let us know if your use case is not covered.
 
 .. _getting-help-containers:
 
diff --git a/doc/sphinx-guides/source/container/running/production.rst b/doc/sphinx-guides/source/container/running/production.rst
@@ -1,27 +1,21 @@
-Production (Future)
-===================
+Production
+==========
 
 .. contents:: |toctitle|
 	:local:
 
 Status
 ------
 
-The images described in this guide are not yet recommended for production usage, but we think we are close. (Tagged releases are done; see the "supported image tags" section for :ref:`Application <app-image-supported-tags>` and :ref:`Config Baker <config-image-supported-tags>` images.) For now, please see :doc:`demo`.
+As of Dataverse 6.8, when we introduced image tagging per version (see the :ref:`app-image-supported-tags` section for the :ref:`application image <app-image-supported-tags>`), we feel that the images described in this guide are ready for production use. Enjoy!
 
-We'd like to make the following improvements:
+The images and the documentation is not perfect, of course.
 
-- More docs on setting up additional features
+For now, we recommend following the :doc:`demo` tutorial. It will help you learn how to configure and secure your installation. Not that instead of "latest" you might want to select a specific version. Again see :ref:`app-image-supported-tags`.
 
-  - How to set up Rserve.
+The Dataverse guides were originally written with a non-Docker installation in mind so we'd like rewrite them with both Docker and non-Docker in mind. This is a big job, obviously. 😅 We know we'd like to write more about ports. We'd like to explain `how to set up Rserve <https://github.com/IQSS/dataverse/issues/11731>`_. Etc., etc.
 
-- Go through all the features in docs and check what needs to be done differently with containers
-
-  - Check ports, for example.
-
-To join the discussion on what else might be needed before declaring images ready for production, please comment on https://dataverse.zulipchat.com/#narrow/stream/375812-containers/topic/containers.20for.20production/near/434979159
-
-You are also very welcome to join our meetings. See "how to help" below.
+To talk about your ideas for making the images and docs better for production, please feel free to join the `containers for production <https://dataverse.zulipchat.com/#narrow/channel/375812-containers/topic/containers.20for.20production/with/451611258>`_ topic or join a working group meeting (see :ref:`helping-containers`).
 
 Limitations
 -----------
@@ -31,9 +25,9 @@ Limitations
 How to Help
 -----------
 
-You can help the effort to support these images in production by trying them out (see :doc:`demo`) and giving feedback (see :ref:`helping-containers`).
+Please try the images (see :doc:`demo`) and give feedback (see :ref:`helping-containers`)! ❤️
 
 Alternatives
 ------------
 
-Until the images are ready for production, please use the traditional installation method described in the :doc:`/installation/index`.
+The traditional (non-Docker) installation method is described in the :doc:`/installation/index`.
diff --git a/doc/sphinx-guides/source/contributor/documentation.md b/doc/sphinx-guides/source/contributor/documentation.md
@@ -66,7 +66,7 @@ From a terminal, switch to the "dataverse" directory you just cloned. This is th
 
 Then try running this command:
 
-`docker run -it --rm -v $(pwd):/docs sphinxdoc/sphinx:7.2.6 bash -c "cd doc/sphinx-guides && pip3 install -r requirements.txt && make html"`
+`docker run -it --rm -v $(pwd):/docs sphinxdoc/sphinx:7.4.0 bash -c "cd doc/sphinx-guides && pip3 install -r requirements.txt && make html"`
 
 If all goes well, you should be able to open `doc/sphinx-guides/build/html/index.html` to see the guides you just built.
 
@@ -187,7 +187,7 @@ The HTML version of the guides is the official one. Any other formats are mainta
 
 If you would like to build a PDF version of the guides and have Docker installed, please try the command below from the root of the git repo:
 
-`docker run -it --rm -v $(pwd):/docs sphinxdoc/sphinx-latexpdf:7.2.6 bash -c "cd doc/sphinx-guides && pip3 install -r requirements.txt && make latexpdf LATEXMKOPTS=\"-interaction=nonstopmode\"; cd ../.. && ls -1 doc/sphinx-guides/build/latex/Dataverse.pdf"`
+`docker run -it --rm -v $(pwd):/docs sphinxdoc/sphinx-latexpdf:7.4.0 bash -c "cd doc/sphinx-guides && pip3 install -r requirements.txt && make latexpdf LATEXMKOPTS=\"-interaction=nonstopmode\"; cd ../.. && ls -1 doc/sphinx-guides/build/latex/Dataverse.pdf"`
 
 A few notes about the command above:
 
diff --git a/doc/sphinx-guides/source/installation/config.rst b/doc/sphinx-guides/source/installation/config.rst
@@ -556,6 +556,8 @@ dataverse.pid.*.datacite.username
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 dataverse.pid.*.datacite.password
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+dataverse.feature.only-update-datacite-when-needed
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 PID Providers of type ``datacite`` require four additional parameters that define how the provider connects to DataCite.
 DataCite has two APIs that are used in Dataverse:
@@ -571,6 +573,11 @@ for `Fabrica <https://doi.datacite.org/>`_ and their APIs. You need to provide
 the same credentials (``username``, ``password``) to Dataverse software to mint and manage DOIs for you.
 As noted above, you should use one of the more secure options for setting the password.
 
+The `only-update-datacite-when-needed feature` flag is a global option that causes Dataverse to GET the latest metadata from DataCite
+for a DOI and compare it with the current metadata in Dataverse and only sending a following POST request if needed. This potentially
+substitutes a read for an unnecessary write at DataCite, but would result in extra reads when all metadata in Dataverse is new. 
+Setting the flag to "true" is recommended when using DataCite file DOIs.
+
 CrossRef-specific Settings
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -3843,6 +3850,9 @@ please find all known feature flags below. Any of these flags can be activated u
     * - role-assignment-history
       - Turns on tracking/display of role assignments and revocations for collections, datasets, and files
       - ``Off``
+    * - only-update-datacite-when-needed
+      - Only contact DataCite to update a DOI after checking to see if DataCite has outdated information (for efficiency, lighter load on DataCite, especially when using file DOIs).
+      - ``Off``
 
 **Note:** Feature flags can be set via any `supported MicroProfile Config API source`_, e.g. the environment variable
 ``DATAVERSE_FEATURE_XXX`` (e.g. ``DATAVERSE_FEATURE_API_SESSION_AUTH=1``). These environment variables can be set in your shell before starting Payara. If you are using :doc:`Docker for development </container/dev-usage>`, you can set them in the `docker compose <https://docs.docker.com/compose/environment-variables/set-environment-variables/>`_ file.
diff --git a/doc/sphinx-guides/source/installation/intro.rst b/doc/sphinx-guides/source/installation/intro.rst
@@ -17,16 +17,18 @@ Jump ahead to :doc:`config` or :doc:`upgrading` for an existing Dataverse instal
 Intended Audience
 -----------------
 
-This guide is intended primarily for sysadmins who are installing, configuring, and upgrading a Dataverse installation. 
+This guide is intended primarily for sysadmins who are installing, configuring, and upgrading a Dataverse installation. This guide was written with non-Docker installation in mind but if you're interested in Docker, see the :doc:`/container/index`.
 
 Sysadmins are expected to be comfortable using standard Linux commands, issuing ``curl`` commands, and running SQL scripts.
 
 Related Guides
 --------------
 
+See the :doc:`/container/index` if you want to run Dataverse in Docker.
+
 Many "admin" functions can be performed by Dataverse installation users themselves (non-superusers) as documented in the :doc:`/user/index` and that guide is a good introduction to the features of the Dataverse Software from an end user perspective.
 
-If you are a sysadmin who likes to code, you may find the :doc:`/api/index` useful, and you may want to consider improving the installation script or hacking on the community-lead configuration management options mentioned in the :doc:`prep` section. If you **really** like to code and want to help with the Dataverse Software code, please check out the :doc:`/developers/index`!
+If you are a sysadmin who likes to code, you may find the :doc:`/api/index` useful, and you may want to consider improving the installation script or hacking on the community-lead configuration management options mentioned in the :doc:`prep` section. If you **really** like to code and want to help with the Dataverse code or documentation, please check out the :doc:`/contributor/index` and the :doc:`/developers/index`!
 
 .. _support:
 
diff --git a/doc/sphinx-guides/source/installation/prep.rst b/doc/sphinx-guides/source/installation/prep.rst
@@ -19,6 +19,13 @@ Standard Installation
 
 Installing the Dataverse Software involves some system configuration followed by executing an installation script that will guide you through the installation process as described in :doc:`installation-main`, but reading about the :ref:`architecture` of the Dataverse Software is recommended first.
 
+.. _docker-installation:
+
+Docker Installation
++++++++++++++++++++
+
+See the :doc:`/container/index`, especially :doc:`/container/running/index` and :doc:`/container/running/production`.
+
 .. _advanced:
 
 Advanced Installation
@@ -32,7 +39,7 @@ There are some community-lead projects to use configuration management tools suc
 
 (Please note that the "dataverse-ansible" repo is used in a script that allows the Dataverse Software to be installed on Amazon Web Services (AWS) from arbitrary GitHub branches as described in the :doc:`/developers/deployment` section of the Developer Guide.)
 
-The Dataverse Project team is happy to "bless" additional community efforts along these lines (i.e. Docker, Chef, Salt, etc.) by creating a repo under https://github.com/gdcc and managing team access.
+The Dataverse Project team is happy to "bless" additional community efforts along these lines (i.e. Chef, Salt, etc.) by creating a repo under https://github.com/gdcc and managing team access.
 
 The Dataverse Software permits a fair amount of flexibility in where you choose to install the various components. The diagram below shows a load balancer, multiple proxies and web servers, redundant database servers, and offloading of potentially resource intensive work to a separate server. (Glassfish is shown rather than Payara.)
 
diff --git a/modules/container-base/README.md b/modules/container-base/README.md
@@ -13,18 +13,13 @@ this image for other purposes than the Dataverse application.
 
 ## Quick Reference
 
-**Maintained by:** 
-
-This image is created, maintained and supported by the Dataverse community on a best-effort basis.
-
 **Where to find documentation:**
 
 The [Dataverse Container Guide - Base Image](https://guides.dataverse.org/en/latest/container/base-image.html)
 provides in-depth information about content, building, tuning and so on for this image. 
 
 **Where to get help and ask questions:**
 
-IQSS will not offer support on how to deploy or run it. Please reach out to the community for help on using it.
 You can join the Community Chat at https://chat.dataverse.org and https://groups.google.com/g/dataverse-community
 to ask for help and guidance.
 
diff --git a/modules/container-configbaker/README.md b/modules/container-configbaker/README.md
@@ -7,18 +7,13 @@ You may use this image as is, base your own derivative image on it or use bind m
 
 ## Quick Reference
 
-**Maintained by:** 
-
-This image is created, maintained and supported by the Dataverse community on a best-effort basis.
-
 **Where to find documentation:**
 
 The [Dataverse Container Guide - Config Baker Image](https://guides.dataverse.org/en/latest/container/configbaker-image.html)
 provides information about this image. 
 
 **Where to get help and ask questions:**
 
-IQSS will not offer support on how to deploy or run it. Please reach out to the community for help on using it.
 You can join the Community Chat at https://chat.dataverse.org and https://groups.google.com/g/dataverse-community
 to ask for help and guidance.
 
diff --git a/src/main/java/edu/harvard/iq/dataverse/pidproviders/doi/datacite/DOIDataCiteRegisterService.java b/src/main/java/edu/harvard/iq/dataverse/pidproviders/doi/datacite/DOIDataCiteRegisterService.java
@@ -95,7 +95,14 @@ public String reRegisterIdentifier(String identifier, Map<String, String> metada
             }
             retString = "metadata:\\r" + client.postMetadata(xmlMetadata) + "\\r";
         }
-        if (!target.equals(client.getUrl(numericIdentifier))) {
+        String currentUrl = null;
+        try {
+            //May get a 204 if the DOI is still draft
+            currentUrl = client.getUrl(numericIdentifier);
+        } catch (RuntimeException ex) {
+            logger.fine("Error getting Url for " + numericIdentifier + ": " + ex.getMessage());
+        }
+        if (!target.equals(currentUrl)) {
             logger.info("Updating target URL to " +  target);
             client.postUrl(numericIdentifier, target);
             retString = retString + "url:\\r" + target;
diff --git a/src/main/java/edu/harvard/iq/dataverse/pidproviders/doi/datacite/DataCiteDOIProvider.java b/src/main/java/edu/harvard/iq/dataverse/pidproviders/doi/datacite/DataCiteDOIProvider.java
@@ -15,6 +15,7 @@
 import edu.harvard.iq.dataverse.FileMetadata;
 import edu.harvard.iq.dataverse.GlobalId;
 import edu.harvard.iq.dataverse.pidproviders.doi.AbstractDOIProvider;
+import edu.harvard.iq.dataverse.settings.FeatureFlags;
 import edu.harvard.iq.dataverse.util.json.JsonUtil;
 import jakarta.json.JsonObject;
 
@@ -217,7 +218,11 @@ public boolean publicizeIdentifier(DvObject dvObject) {
         metadata.put("datacite.publicationyear", generateYear(dvObject));
         metadata.put("_target", getTargetUrl(dvObject));
         try {
-            doiDataCiteRegisterService.registerIdentifier(identifier, metadata, dvObject);
+            if (FeatureFlags.ONLY_UPDATE_DATACITE_WHEN_NEEDED.enabled()) {
+                doiDataCiteRegisterService.reRegisterIdentifier(identifier, metadata, dvObject);
+            } else {
+                doiDataCiteRegisterService.registerIdentifier(identifier, metadata, dvObject);
+            }
             return true;
         } catch (Exception e) {
             logger.log(Level.WARNING, "modifyMetadata failed: " + e.getMessage(), e);
diff --git a/src/main/java/edu/harvard/iq/dataverse/pidproviders/doi/datacite/DataCiteRESTfullClient.java b/src/main/java/edu/harvard/iq/dataverse/pidproviders/doi/datacite/DataCiteRESTfullClient.java
diff --git a/src/main/java/edu/harvard/iq/dataverse/settings/FeatureFlags.java b/src/main/java/edu/harvard/iq/dataverse/settings/FeatureFlags.java

-Original file line number
+Diff line change
           overwrite: true
       - name: Execute payara war deployment remotely
 -        uses: appleboy/[email protected].2
 +        uses: appleboy/[email protected].3
         env:
           INPUT_WAR_FILE: ${{ env.war_file }}
         with: