IQSS/8914 COAR compliant LDN messaging (#10490)

* HDC 3b 1.5 initial update

* add workflow settings to main list per qa

* start adding default bestID

* replace system.out.println call

* Improve default compound field handling and some reformatting

* add template custom instructions info

* get blocks from metadataroot

* fix parsing to match spec/dash msg

* use getString for 'Relationship', revert other changes

* temporarily drop name/type in display

* misplaced }

* Add callback loop and make display tolerant wrt name/type being found

* add fields missing in https://notify.coar-repositories.org/scenarios/10/

* debugging

* added logging/switched to info for HDC debugging

* merge issues

* updates to send context, use as namespace, use relationType, DASH-NRS

* doc updates

* note some ToDos

* New format has strings, not objects with @id

* add space to fix link

* more doc tweaks

* refactor, use JVMSettings, add tests, add superusers only flag

* refactor wf step - name change, multiple targets, gen. improvements

prioritize publicationIDNumber field over url, support all IDTypes if
they provide URIs, check DOIs and Handles for any form, including raw,
fix logic for other fields, check for URI form in all cases, handle null
results, ...

* drop lower case start to DataCite relation types, use constants

* test fixes, use COAR step announcement builder

* don't autofollow redirect, handle more redirect codes

* get absolute redirect url, fix signposting parsing

* remove overloaded method (failing in xhtml)

* cleanup, fine logging

* info to fine

* doc updates

* Changes per review

* Apply suggestions from code review

Co-authored-by: Philip Durbin <[email protected]>

* typo

* add IT tests to list

* null bug fix, refactor tests so that allowed hosts can be mocked

* update docs per QA

* missing /

---------

Co-authored-by: Philip Durbin <[email protected]>

Author: qqmyers

doc/release-notes/10490-COAR-Notify.md

@@ -0,0 +1,23 @@
+### Support for COAR Notify Relationship Announcement
+
+Dataverse now supports sending and receiving [Linked Data Notification ](https://www.w3.org/TR/ldn/) messages involved in the 
+[COAR Notify Relationship Announcement Workflow](https://coar-notify.net/catalogue/workflows/repository-relationship-repository/).
+
+Dataverse can send messages to configured repositories announcing that a dataset has a related publication (as defined in the dataset metadata). This may be done automatically upon publication or triggered manually by a superuser. The receiving repository may do anything with the message, with the default expectation being that the repository will create a backlink from the publication to the dataset (assuming the publication exists in the repository, admins agree the link makes sense, etc.)
+
+Conversely, Dataverse can receive notices from other configured repositories announcing relationships between their publications and datasets. If the referenced dataset exists in the Dataverse instance, a notification will be sent to users who can publish the dataset, or, optionally, only superusers who can publish the dataset. They can then decide whether to create a backlink to the publication in the dataset metadata.
+
+(Earlier releases of Dataverse had experimental support in this area that was based on message formats defined prior to finalization of the COAR Notify specification for relationship announcements.)
+
+#### New Settings/JVM Options
+
+Configuration for sending messages involves specifying the
+:COARNotifyRelationshipAnnouncementTargets and :COARNotifyRelationshipAnnouncementTriggerFields
+
+Configuration to receive messages involves specifying
+DATAVERSE_LDN_ALLOWED_HOSTS (dataverse.ldn.allowed-hosts)
+
+Notifications are sent by default to users who can publish a dataset. The option below can be used to restrict notifications to superusers who can publish the dataset.
+ 
+DATAVERSE_COAR_NOTIFY_RELATIONSHIP_ANNOUNCEMENT_NOTIFY_SUPERSUSERS_ONLY
+

doc/sphinx-guides/source/api/linkeddatanotification.rst

@@ -1,12 +1,17 @@
 Linked Data Notification API
 ============================
 
-Dataverse has a limited, experimental API implementing a Linked Data Notification inbox allowing it to receive messages indicating a link between an external resource and a Dataverse dataset.
+Dataverse has an API implementing a Linked Data Notification (LDN) inbox allowing it to receive messages implementing the `COAR Notify Relationship Announcement <https://coar-notify.net/catalogue/workflows/repository-relationship-repository/>`_ indicating a link between an external resource and a Dataverse dataset.
+
+Dataverse has a related capability to send COAR Notify Relationship Announcement messages, automatically upon publication or manually. See the :doc:`/developers/workflows` section of the Guides.
+
 The motivating use case is to support a use case where Dataverse administrators may wish to create back-links to the remote resource (e.g. as a Related Publication, Related Material, etc.).
 
-Upon receipt of a relevant message, Dataverse will create Announcement Received notifications for superusers, who can edit the dataset involved. (In the motivating use case, these users may then add an appropriate relationship and use the Update Curent Version publishing option to add it to the most recently published version of the dataset.)
+Upon receipt of a relevant message, Dataverse will create Announcement Received notifications for users who can edit the dataset involved. Notifications can be restricted to superusers who can publish the dataset as described below. (In the motivating use case, these superusers may then add an appropriate relationship and use the Update Curent Version publishing option to add it to the most recently published version of the dataset.)
+
+The ``dataverse.ldn.allowed-hosts`` JVM option is a comma-separated list of hosts from which Dataverse will accept and process messages. By default, no hosts are allowed. ``*`` can be used in testing to indicate all hosts are allowed.
 
-The ``:LDNMessageHosts`` setting is a comma-separated whitelist of hosts from which Dataverse will accept and process messages. By default, no hosts are allowed. ``*`` can be used in testing to indicate all hosts are allowed.
+The ``dataverse.ldn.coar-notify.relationship-announcement.notify-superusers-only`` JVM option can be set to ``true`` to restrict notifications to superusers only (those who can publish the dataset). The default is to notify all users who can publish the dataset.
 
 Messages can be sent via POST, using the application/ld+json ContentType:
 
@@ -15,10 +20,12 @@ Messages can be sent via POST, using the application/ld+json ContentType:
   export SERVER_URL=https://demo.dataverse.org
   
   curl -X POST -H 'ContentType:application/ld+json' $SERVER_URL/api/inbox --upload-file message.jsonld
+  
 
-The supported message format is described by `our preliminary specification <https://docs.google.com/document/d/1dqj8_vEcIBeyDIZCaPQvp0FM1eSGO_5CSNCdXOpoUz0/edit?usp=sharing>`_. The format is expected to change in the near future to match the standard for relationship announcements being developed as part of `the COAR Notify Project <https://notify.coar-repositories.org/>`_. 
+The supported message format is described by `the COAR Notify Relationship Announcement specification <https://coar-notify.net/catalogue/workflows/repository-relationship-repository/2/>`_. 
 
-An example message is shown below. It indicates that a resource with the name "An Interesting Title" exists and "IsSupplementedBy" the dataset with DOI https://doi.org/10.5072/FK2/GGCCDL. If this dataset is managed in the receiving Dataverse, a notification will be sent to user with the relevant permissions (as described above).
+An example message is shown below. It indicates that a resource in the "Harvard DASH" test server has, as a "supplement", the dataset with DOI doi:10.5074/FKNOAHNQ.
+If this dataset is managed in the receiving Dataverse, a notification will be sent to user with the relevant permissions (as described above).
 
 .. code:: json
 
@@ -27,39 +34,44 @@ An example message is shown below. It indicates that a resource with the name "A
       "https://www.w3.org/ns/activitystreams",
       "https://purl.org/coar/notify"
     ],
-    "id": "urn:uuid:94ecae35-dcfd-4182-8550-22c7164fe23f",
     "actor": {
-      "id": "https://research-organisation.org/dspace",
-      "name": "DSpace Repository",
+      "id": "https://harvard-dash.staging.4science.cloud",
+      "name": "Harvard DASH",
       "type": "Service"
     },
     "context": {
-      "IsSupplementedBy":
-        {
-          "id": "http://dev-hdc3b.lib.harvard.edu/dataset.xhtml?persistentId=doi:10.5072/FK2/GGCCDL",
-          "ietf:cite-as": "https://doi.org/10.5072/FK2/GGCCDL",
-          "type": "sorg:Dataset"
-        }
+      "id": "https://harvard-dash.staging.4science.cloud/handle/1/42718322",
+      "ietf:cite-as": "https://harvard-dash.staging.4science.cloud/handle/1/42718322",
+      "ietf:item": {
+        "id": "https://harvard-dash.staging.4science.cloud/bitstreams/e2ae80a1-35e5-411b-9ef1-9175f6cccf23/download",
+        "mediaType": "application/pdf",
+        "type": [
+          "Article",
+          "sorg:ScholarlyArticle"
+        ]
+      },
+      "type": "sorg:AboutPage"
     },
+    "id": "urn:uuid:3c933c09-c246-473d-bea4-674db168cfee",
     "object": {
-      "id": "https://research-organisation.org/dspace/item/35759679-5df3-4633-b7e5-4cf24b4d0614",
-      "ietf:cite-as": "https://research-organisation.org/authority/resolve/35759679-5df3-4633-b7e5-4cf24b4d0614",
-      "sorg:name": "An Interesting Title",
-      "type": "sorg:ScholarlyArticle"
+      "as:object": "doi: 10.5074/FKNOAHNQ",
+      "as:relationship": "http://purl.org/vocab/frbr/core#supplement",
+      "as:subject": "https://harvard-dash.staging.4science.cloud/handle/1/42718322",
+      "id": "urn:uuid:0851f805-c52f-4d0b-81ac-a07e99c33e20",
+      "type": "Relationship"
     },
     "origin": {
-      "id": "https://research-organisation.org/dspace",
-      "inbox": "https://research-organisation.org/dspace/inbox/",
+      "id": "https://harvard-dash.staging.4science.cloud",
+      "inbox": "https://harvard-dash.staging.4science.cloud/server/ldn/inbox",
       "type": "Service"
     },
     "target": {
-      "id": "https://research-organisation.org/dataverse",
-      "inbox": "https://research-organisation.org/dataverse/inbox/",
+      "id": "http://ec2-3-238-245-253.compute-1.amazonaws.com/",
+      "inbox": "http://ec2-3-238-245-253.compute-1.amazonaws.com/api/inbox",
       "type": "Service"
     },
     "type": [
       "Announce",
-      "coar-notify:ReleaseAction"
+      "coar-notify:RelationshipAction"
     ]
   }
-

doc/sphinx-guides/source/api/native-api.rst

@@ -8241,7 +8241,7 @@ Get details of a workflow with a given id::
 
    GET http://$SERVER/api/admin/workflows/$id
 
-Add a new workflow. Request body specifies the workflow properties and steps in JSON format.
+Add a new workflow. Request body specifies the workflow properties and steps in JSON format. Specifically, the body of the message should be a JSON Object with a String "name" for the workflow and a "steps" JSON Array containing a JSON Object per workflow step. (See :doc:`/developers/workflows` for the exiting steps and their required JSON representations.)
 Sample ``json`` files are available at ``scripts/api/data/workflows/``::
 
    POST http://$SERVER/api/admin/workflows

doc/sphinx-guides/source/developers/workflows.rst

@@ -202,30 +202,30 @@ Note - the example step includes two settings required for any archiver, three (
   }
 
 
-ldnannounce
-+++++++++++
+coarNotifyRelationshipAnnouncement
+++++++++++++++++++++++++++++++++++
 
-An experimental step that sends a Linked Data Notification (LDN) message to a specific LDN Inbox announcing the publication/availability of a dataset meeting certain criteria. 
+A step that sends a `COAR Notify Relationship Announcement <https://coar-notify.net/catalogue/workflows/repository-relationship-repository/>`_ message, using the `Linked Data Notification (LDN) <https://www.w3.org/TR/ldn/>`_ message standard,
+to a specific set of LDN Inboxes announcing a relationship between a newly published/available dataset and an external resource (e.g. one managed by the recipient). 
 
 The two parameters are
-* ``:LDNAnnounceRequiredFields`` - a list of metadata fields that must exist to trigger the message. Currently, the message also includes the values for these fields but future versions may only send the dataset's persistent identifier (making the receiver responsible for making a call-back to get any metadata).
-* ``:LDNTarget`` - a JSON object containing an ``inbox`` key whose value is the URL of the target LDN inbox to which messages should be sent, e.g. ``{"id": "https://dashv7-dev.lib.harvard.edu","inbox": "https://dashv7-api-dev.lib.harvard.edu/server/ldn/inbox","type": "Service"}`` ).
-
-The supported message format is desribed by `our preliminary specification <https://docs.google.com/document/d/1dqj8_vEcIBeyDIZCaPQvp0FM1eSGO_5CSNCdXOpoUz0/edit?usp=sharing>`_. The format is expected to change in the near future to match the standard for relationship announcements being developed as part of `the COAR Notify Project <https://notify.coar-repositories.org/>`_. 
+* ``:COARNotifyRelationshipAnnouncementTriggerFields`` - a list of metadata field types that can trigger messages. Separate messages will be sent for each field (whether of the same type or not).
+* ``:COARNotifyRelationshipAnnouncementTargets`` - a JSON Array of JSON objects containing ``id``, ``inbox``, and ``type`` fields as required by the `COAR Notify Relationship Announcement specification <https://coar-notify.net/catalogue/workflows/repository-relationship-repository/2/>`_ .
+The ``inbox`` value should be the full URL of the target LDN inbox to which messages should be sent, e.g. ``{"id": "https://dashv7-dev.lib.harvard.edu","inbox": "https://dashv7-api-dev.lib.harvard.edu/server/ldn/inbox","type": ["Service"]}`` ).
 
 
 .. code:: json
 
 
   {
     "provider":":internal",
-    "stepType":"ldnannounce",
+    "stepType":"coarNotifyRelationshipAnnouncement",
     "parameters": {
-      "stepName":"LDN Announce"
+      "stepName":"COAR Notify Relationship Announcement"
     },
     "requiredSettings": {
-      ":LDNAnnounceRequiredFields": "string",
-      ":LDNTarget": "string"
+      ":COARNotifyRelationshipAnnouncementTriggerFields": "string",
+      ":COARNotifyRelationshipAnnouncementTargets": "string"
     }
   }
 

doc/sphinx-guides/source/installation/config.rst

@@ -3716,7 +3716,7 @@ Experimental. See :doc:`/developers/search-services`.
 .. _dataverse.cors:
 
 CORS Settings
--------------
++++++++++++++
 
 The following settings control Cross-Origin Resource Sharing (CORS) for your Dataverse installation.
 
@@ -3798,6 +3798,23 @@ Example: ``dataverse.api.mdc.min-delay-ms=100`` (enforces a minimum 100ms delay
 
 Can also be set via any `supported MicroProfile Config API source`_, e.g. the environment variable ``DATAVERSE_API_MDC_MIN_DELAY_MS``.
 
+.. dataverse.ldn
+
+Linked Data Notifications (LDN) Allowed Hosts
++++++++++++++++++++++++++++++++++++++++++++++
+
+Dataverse supports receiving LDN notifications via the /api/inbox endpoint. The dataverse.ldn.allowed-hosts allows you to specify the list of host IP addresses from which LDN notifications can be received, or ``*`` to receive messages from anywhere.
+
+Example: ``dataverse.ldn.allowed-hosts=*``
+
+COAR Notify Relationship Announcement Notify Superusers Only
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+When Dataverse receives an LDN message conforming to the COAR Notify Relationship Announcement format and the message is about a dataset hosted in the installation, Dataverse will send an notification to users who have permission to publish the dataset.
+This can instead be restricted to only superusers who can publish the dataset using this option.
+
+Example: ``dataverse.coar-notify.relationship-announcement.notify-superusers-only=true``
+
 .. _feature-flags:
 
 Feature Flags

scripts/api/data/workflows/internal-coar-notify-relationship-announcement-workflow.json

@@ -0,0 +1,16 @@
+{
+  "name": "COAR Notify Relationship Announcement workflow",
+  "steps": [
+    {
+      "provider":":internal",
+      "stepType":"coarNotifyRelationshipAnnouncement",
+      "parameters": {
+        "stepName":"LDN Announce"
+      },
+      "requiredSettings": {
+        ":COARNotifyRelationshipAnnouncementTriggerFields": "string",
+        ":COARNotifyRelationshipAnnouncementTargets": "string"
+      }
+    }
+  ]
+}

scripts/api/data/workflows/internal-ldnannounce-workflow.json

@@ -737,9 +737,9 @@ public String getMessageTextBasedOnNotification(UserNotification userNotificatio
                 Object[] paramArrayDatasetMentioned = {
                         userNotification.getUser().getName(),
                         BrandingUtil.getInstallationBrandName(), 
-                        citingResource.getString("@type"),
+                        citingResource.getString("@type", "External Resource"),
                         citingResource.getString("@id"),
-                        citingResource.getString("name"),
+                        citingResource.getString("name", citingResource.getString("@id")),
                         citingResource.getString("relationship"), 
                         systemConfig.getDataverseSiteUrl(),
                         dataset.getGlobalId().toString(),