Webhook features and documentation (#1596)

Signed-off-by: tdruez <[email protected]>

Author: tdruez

CHANGELOG.rst

@@ -1,6 +1,18 @@
 Changelog
 =========
 
+v34.9.6 (unreleased)
+--------------------
+
+- Refine and document the Webhook system
+  https://github.com/aboutcode-org/scancode.io/issues/1587
+  * Add UI to add/delete Webhooks from the project settings
+  * Add a new `add-webhook` management command
+  * Add a `add_webhook` REST API action
+  * Add a new `SCANCODEIO_GLOBAL_WEBHOOK` setting
+  * Add a new chapter dedicated to Webhooks management in the documentation
+  * Add support for custom payload dedicated to Slack webhooks
+
 v34.9.5 (2025-02-19)
 --------------------
 
@@ -15,6 +27,10 @@ v34.9.5 (2025-02-19)
   with other aboutcode submodules.
   https://github.com/aboutcode-org/scancode.io/issues/1423
 
+- Add a ``add-webhook`` management command that allows to add webhook subscription on
+  a project.
+  https://github.com/aboutcode-org/scancode.io/issues/1587
+
 - Add proper progress logging for the ``assemble`` section of the
   ``scan_for_application_packages``.
   https://github.com/aboutcode-org/scancode.io/issues/1601

docs/application-settings.rst

@@ -268,6 +268,54 @@ The web server can be started in DEBUG mode with:
 
     $ SCANCODEIO_LOG_LEVEL=DEBUG make run
 
+.. _scancodeio_settings_site_url:
+
+SCANCODEIO_SITE_URL
+^^^^^^^^^^^^^^^^^^^
+
+The base URL of the ScanCode.io application instance.
+This setting is **required** to generate absolute URLs referencing objects within the
+application, such as in webhook notifications.
+
+The value should be a fully qualified URL, including the scheme (e.g., ``https://``).
+
+Example configuration in the ``.env`` file::
+
+    SCANCODEIO_SITE_URL=https://scancode.example.com/
+
+Default: ``""`` (empty)
+
+.. _scancodeio_settings_global_webhook:
+
+SCANCODEIO_GLOBAL_WEBHOOK
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This setting defines a **global webhook** that will be automatically added as a
+``WebhookSubscription`` for each new project.
+
+The webhook is configured as a dictionary and must include a ``target_url``.
+Additional options control when the webhook is triggered and what data is included
+in the payload.
+
+Example configuration in the ``.env`` file::
+
+    SCANCODEIO_GLOBAL_WEBHOOK=target_url=https://webhook.url,trigger_on_each_run=False,include_summary=True,include_results=False
+
+The available options are:
+
+- ``target_url`` (**required**): The URL where the webhook payload will be sent.
+- ``trigger_on_each_run`` (**default**: ``False``): If ``True``, the webhook is triggered
+  on every pipeline run.
+- ``include_summary`` (**default**: ``False``): If ``True``, a summary of the pipeline
+  run results is included in the payload.
+- ``include_results`` (**default**: ``False``): If ``True``, detailed scan results
+  are included in the payload.
+
+If this setting is provided, ScanCode.io will create a webhook subscription
+**only for newly created projects that are not clones**.
+
+Default: ``{}`` (no global webhook is set)
+
 TIME_ZONE
 ^^^^^^^^^
 

docs/command-line-interface.rst

@@ -344,6 +344,47 @@ add the docker pipeline to your project::
     ``--pipeline map_deploy_to_develop:Java,JavaScript``
 
 
+.. _cli_add_webhook:
+
+`$ scanpipe add-webhook --project PROJECT TARGET_URL`
+-----------------------------------------------------
+
+Adds a webhook subscription to a project.
+
+Required arguments:
+
+- ``target-url``
+  The target URL to which the webhook should send POST requests.
+
+Optional arguments:
+
+- ``--trigger-on-each-run``
+  Trigger the webhook after each individual pipeline run.
+
+- ``--include-summary``
+  Include summary data in the payload.
+
+- ``--include-results``
+  Include results data in the payload.
+
+- ``--inactive``
+  Create the webhook but set it as inactive.
+
+Example usage:
+
+1. Add an active webhook that triggers after each pipeline run::
+
+   $ scanpipe add-webhook my_project https://example.com/webhook --trigger-on-each-run
+
+2. Add a webhook that includes summary and results data::
+
+   $ scanpipe add-webhook my_project https://example.com/webhook --include-summary --include-results
+
+3. Add an inactive webhook::
+
+   $ scanpipe add-webhook my_project https://example.com/webhook --inactive
+
+
 `$ scanpipe execute --project PROJECT`
 --------------------------------------
 

docs/faq.rst

@@ -205,6 +205,21 @@ Also, A new GitHub action is available at
 `scancode-action repository <https://github.com/nexB/scancode-action>`_
 to run ScanCode.io pipelines from your GitHub Workflows.
 
+How can I get notified about my project progression?
+-----------------------------------------------------
+
+You can monitor your project's progress in multiple ways:
+
+- **User Interface:** The project details page provides real-time updates on pipeline
+  execution.
+- **REST API:** Use the API to programmatically check the status of your projects.
+- **CLI Monitoring:** The ``scanpipe list-projects`` command provides an overview of
+  project states.
+- **Webhook Integration:** You can set up webhooks to receive updates in your preferred
+  notification system. For more details, refer to the :ref:`webhooks` section.
+- **Slack notifications:** Get project updates directly in Slack by configuring an
+  incoming webhook. See :ref:`webhooks_slack_notifications` for setup instructions.
+
 .. _faq_tag_input_files:
 
 How to tag input files?

docs/index.rst

@@ -50,6 +50,7 @@ you’ll find information on:
     command-line-interface
     rest-api
     automation
+    webhooks
     application-settings
     distros-os-images
 

docs/project-configuration.rst

@@ -104,7 +104,6 @@ Default is ``None``, in which case all files will be scanned.
     set using the .env file, and the project setting ``scan_max_file_size`` takes
     precedence over the scancodeio setting ``SCANCODEIO_SCAN_MAX_FILE_SIZE``.
 
-
 ignored_patterns
 ^^^^^^^^^^^^^^^^
 

docs/rest-api.rst

@@ -198,6 +198,81 @@ about the project, including the status of any pipeline run:
         "created_date": "2021-07-21T16:06:29.132795+02:00"
     }
 
+.. _rest_api_webhooks:
+
+Adding webhooks
+^^^^^^^^^^^^^^^
+
+When creating a project, you can also **register webhook subscriptions** that will
+notify external services about pipeline execution events.
+
+You can either provide:
+ - A **single webhook URL** using the ``webhook_url`` field.
+ - A **list of detailed webhook configurations** using the ``webhooks`` field.
+
+Webhook fields:
+    - ``webhook_url`` (string, optional): A single webhook URL to be notified.
+    - ``webhooks`` (list, optional): A list of webhook configurations.
+        - ``target_url`` (string, required): The URL to which the webhook will send a
+          ``POST`` request.
+        - ``trigger_on_each_run`` (boolean, optional): If ``true``, the webhook will be
+          triggered after each individual pipeline run. Default is ``false``.
+        - ``include_summary`` (boolean, optional): If ``true``, the webhook payload
+          will include the summary data of the pipeline execution. Default is ``false``.
+        - ``include_results`` (boolean, optional): If ``true``, the webhook payload
+          will include the full results data of the pipeline execution.
+          Default is ``false``.
+        - ``is_active`` (boolean, optional): If ``true``, the webhook is active and
+          will be triggered when the specified conditions are met. Default is ``true``.
+
+Using cURL to register a webhook:
+
+.. code-block:: console
+
+    api_url="http://localhost/api/projects/"
+    content_type="Content-Type: application/json"
+    data='{
+        "name": "project_name",
+        "webhook_url": "https://example.com/webhook"
+    }'
+
+    curl -X POST "$api_url" -H "$content_type" -d "$data"
+
+.. code-block:: json
+
+    {
+        "name": "project_name",
+        "webhook_url": "https://example.com/webhook"
+    }
+
+Using cURL to register multiple webhooks:
+
+.. code-block:: console
+
+    api_url="http://localhost/api/projects/"
+    content_type="Content-Type: application/json"
+    data='{
+        "name": "project_name",
+        "webhooks": [
+            {
+                "target_url": "https://example.com/webhook1",
+                "trigger_on_each_run": true,
+                "include_summary": false,
+                "include_results": true,
+                "is_active": true
+            },
+            {
+                "target_url": "https://example.com/webhook2",
+                "trigger_on_each_run": false,
+                "include_summary": true,
+                "include_results": false,
+                "is_active": true
+            }
+        ]
+    }'
+
+    curl -X POST "$api_url" -H "$content_type" -d "$data"
+
 Project details
 ---------------
 
@@ -278,6 +353,8 @@ Using cURL to upload a local file:
         "status": "Input(s) added."
     }
 
+.. _rest_api_add_pipeline:
+
 Add pipeline
 ^^^^^^^^^^^^
 
@@ -315,6 +392,58 @@ Using cURL:
         "status": "Pipeline added."
     }
 
+.. _rest_api_add_webhook:
+
+Add webhook
+^^^^^^^^^^^
+
+This action adds a webhook subscription to the ``project``.
+A webhook allows external services to receive real-time notifications about project
+pipeline execution events.
+
+``POST /api/projects/d4ed9405-5568-45ad-99f6-782a9b82d1d2/add_webhook/``
+
+Data:
+    - ``target_url`` (string, required): The URL to which the webhook will send
+      a ``POST`` request when triggered.
+    - ``trigger_on_each_run`` (boolean, optional): If ``true``, the webhook will be
+      triggered after each individual pipeline run. Default is ``false``.
+    - ``include_summary`` (boolean, optional): If ``true``, the webhook payload will
+      include the summary data of the pipeline execution. Default is ``false``.
+    - ``include_results`` (boolean, optional): If ``true``, the webhook payload will
+      include the full results data of the pipeline execution. Default is ``false``.
+    - ``is_active`` (boolean, optional): If ``true``, the webhook is active and will
+      be triggered when the specified conditions are met. Default is ``true``.
+
+Using cURL:
+
+.. code-block:: console
+
+    api_url="http://localhost/api/projects/6461408c-726c-4b70-aa7a-c9cc9d1c9685/add_webhook/"
+    content_type="Content-Type: application/json"
+    data='{
+        "target_url": "https://example.com/webhook",
+        "trigger_on_each_run": true,
+        "include_summary": true,
+        "include_results": false,
+        "is_active": true
+    }'
+
+    curl -X POST "$api_url" -H "$content_type" -d "$data"
+
+.. code-block:: json
+
+    {
+        "status": "Webhook added."
+    }
+
+.. note::
+    - Webhooks will only be triggered for active subscriptions.
+    - If ``trigger_on_each_run`` is set to ``false``, the webhook will only trigger
+      after all pipeline runs are completed.
+    - The ``include_summary`` and ``include_results`` fields allow customization of
+      the webhook payload.
+
 Archive
 ^^^^^^^
 

docs/user-interface.rst

@@ -117,6 +117,34 @@ your specific requirements.
     Project Settings UI and click on the **Config file** link in the left section under
     **Download**.
 
+.. _user_interface_manage_webhook:
+
+Manage Webhook
+^^^^^^^^^^^^^^
+
+The **Manage Webhook** section in the Project Settings view allows you to configure
+Webhook subscriptions for project pipeline execution events.
+
+From the Settings page, you can:
+
+- **Add a new webhook** by specifying a target URL that will receive event notifications.
+- **Edit an existing webhook** to update its target URL, event triggers, or payload settings.
+- **Delete a webhook** to stop receiving notifications for that project.
+
+Each webhook is associated with a **WebhookSubscription** and includes the following
+configurable options:
+
+- **Target URL**: The destination where the webhook payload will be sent when triggered.
+- **Trigger on Each Run**: Choose whether to send notifications after each pipeline run
+  or only after all runs are completed.
+- **Include Summary**: Decide whether to include summary data in the webhook payload.
+- **Include Results**: Optionally include detailed scan results in the webhook payload.
+- **Active Status**: Enable or disable the webhook without deleting it.
+
+.. note::
+    Ensure the target URL is correctly configured to receive webhook requests and
+    process them accordingly.
+
 Archive a Project
 ^^^^^^^^^^^^^^^^^