Merge remote-tracking branch 'origin/scorecard_integration' into scorecard_integration

Author: 404-geek

CHANGELOG.rst

@@ -1,6 +1,22 @@
 Changelog
 =========
 
+v34.9.6 (unreleased)
+--------------------
+
+- Rename the ``docker``, ``docker_windows``, and ``root_filesystem`` modules to
+  ``analyze_docker``, ``analyze_docker_windows``, and ``analyze_root_filesystem``
+  for consistency.
+
+- Refine and document the Webhook system
+  https://github.com/aboutcode-org/scancode.io/issues/1587
+  * Add UI to add/delete Webhooks from the project settings
+  * Add a new ``add-webhook`` management command
+  * Add a ``add_webhook`` REST API action
+  * Add a new ``SCANCODEIO_GLOBAL_WEBHOOK`` setting
+  * Add a new chapter dedicated to Webhooks management in the documentation
+  * Add support for custom payload dedicated to Slack webhooks
+
 v34.9.5 (2025-02-19)
 --------------------
 
@@ -15,6 +31,14 @@ v34.9.5 (2025-02-19)
   with other aboutcode submodules.
   https://github.com/aboutcode-org/scancode.io/issues/1423
 
+- Add a ``add-webhook`` management command that allows to add webhook subscription on
+  a project.
+  https://github.com/aboutcode-org/scancode.io/issues/1587
+
+- Add proper progress logging for the ``assemble`` section of the
+  ``scan_for_application_packages``.
+  https://github.com/aboutcode-org/scancode.io/issues/1601
+
 v34.9.4 (2025-01-21)
 --------------------
 

aboutcode/pipeline/CHANGELOG.md

@@ -1,5 +1,10 @@
 # Changelog
 
+## Release 0.2.1 (February 24, 2025)
+
+* Include the ``optional_step`` steps in the ``get_graph()`` list.
+  [Issue #1599](https://github.com/aboutcode-org/scancode.io/issues/1599)
+
 ## Release 0.2.0 (November 21, 2024)
 
 * Refactor the ``group`` decorator for pipeline optional steps as ``optional_step``.

aboutcode/pipeline/__init__.py

@@ -31,7 +31,7 @@
 
 module_logger = logging.getLogger(__name__)
 
-__version__ = "0.2.0"
+__version__ = "0.2.1"
 
 
 class PipelineDefinition:
@@ -97,7 +97,7 @@ def get_graph(cls):
                 "doc": getdoc(step),
                 "groups": getattr(step, "groups", []),
             }
-            for step in cls.get_steps()
+            for step in cls.get_steps(groups=cls.get_available_groups())
         ]
 
     @classmethod

docker-compose.yml

@@ -22,7 +22,7 @@ services:
     command: wait-for-it --strict --timeout=60 db:5432 -- sh -c "
         ./manage.py migrate &&
         ./manage.py collectstatic --no-input --verbosity 0 --clear &&
-        gunicorn scancodeio.wsgi:application --bind :8000 --timeout 600 --workers 8 ${GUNICORN_RELOAD_FLAG}"
+        gunicorn scancodeio.wsgi:application --bind :8000 --timeout 600 --workers 8 ${GUNICORN_RELOAD_FLAG:-}"
     env_file:
       - docker.env
     expose:

docs/application-settings.rst

@@ -268,6 +268,54 @@ The web server can be started in DEBUG mode with:
 
     $ SCANCODEIO_LOG_LEVEL=DEBUG make run
 
+.. _scancodeio_settings_site_url:
+
+SCANCODEIO_SITE_URL
+^^^^^^^^^^^^^^^^^^^
+
+The base URL of the ScanCode.io application instance.
+This setting is **required** to generate absolute URLs referencing objects within the
+application, such as in webhook notifications.
+
+The value should be a fully qualified URL, including the scheme (e.g., ``https://``).
+
+Example configuration in the ``.env`` file::
+
+    SCANCODEIO_SITE_URL=https://scancode.example.com/
+
+Default: ``""`` (empty)
+
+.. _scancodeio_settings_global_webhook:
+
+SCANCODEIO_GLOBAL_WEBHOOK
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This setting defines a **global webhook** that will be automatically added as a
+``WebhookSubscription`` for each new project.
+
+The webhook is configured as a dictionary and must include a ``target_url``.
+Additional options control when the webhook is triggered and what data is included
+in the payload.
+
+Example configuration in the ``.env`` file::
+
+    SCANCODEIO_GLOBAL_WEBHOOK=target_url=https://webhook.url,trigger_on_each_run=False,include_summary=True,include_results=False
+
+The available options are:
+
+- ``target_url`` (**required**): The URL where the webhook payload will be sent.
+- ``trigger_on_each_run`` (**default**: ``False``): If ``True``, the webhook is triggered
+  on every pipeline run.
+- ``include_summary`` (**default**: ``False``): If ``True``, a summary of the pipeline
+  run results is included in the payload.
+- ``include_results`` (**default**: ``False``): If ``True``, detailed scan results
+  are included in the payload.
+
+If this setting is provided, ScanCode.io will create a webhook subscription
+**only for newly created projects that are not clones**.
+
+Default: ``{}`` (no global webhook is set)
+
 TIME_ZONE
 ^^^^^^^^^
 

docs/built-in-pipelines.rst

@@ -26,23 +26,23 @@ Pipeline Base Class
 
 Analyse Docker Image
 --------------------
-.. autoclass:: scanpipe.pipelines.docker.Docker()
+.. autoclass:: scanpipe.pipelines.analyze_docker.Docker()
     :members:
     :member-order: bysource
 
 .. _pipeline_analyze_root_filesystem:
 
 Analyze Root Filesystem or VM Image
 -----------------------------------
-.. autoclass:: scanpipe.pipelines.root_filesystem.RootFS()
+.. autoclass:: scanpipe.pipelines.analyze_root_filesystem.RootFS()
     :members:
     :member-order: bysource
 
 .. _analyze_windows_docker_image:
 
 Analyse Docker Windows Image
 ----------------------------
-.. autoclass:: scanpipe.pipelines.docker_windows.DockerWindows()
+.. autoclass:: scanpipe.pipelines.analyze_docker_windows.DockerWindows()
     :members:
     :member-order: bysource
 

docs/command-line-interface.rst

@@ -344,6 +344,47 @@ add the docker pipeline to your project::
     ``--pipeline map_deploy_to_develop:Java,JavaScript``
 
 
+.. _cli_add_webhook:
+
+`$ scanpipe add-webhook --project PROJECT TARGET_URL`
+-----------------------------------------------------
+
+Adds a webhook subscription to a project.
+
+Required arguments:
+
+- ``target-url``
+  The target URL to which the webhook should send POST requests.
+
+Optional arguments:
+
+- ``--trigger-on-each-run``
+  Trigger the webhook after each individual pipeline run.
+
+- ``--include-summary``
+  Include summary data in the payload.
+
+- ``--include-results``
+  Include results data in the payload.
+
+- ``--inactive``
+  Create the webhook but set it as inactive.
+
+Example usage:
+
+1. Add an active webhook that triggers after each pipeline run::
+
+   $ scanpipe add-webhook my_project https://example.com/webhook --trigger-on-each-run
+
+2. Add a webhook that includes summary and results data::
+
+   $ scanpipe add-webhook my_project https://example.com/webhook --include-summary --include-results
+
+3. Add an inactive webhook::
+
+   $ scanpipe add-webhook my_project https://example.com/webhook --inactive
+
+
 `$ scanpipe execute --project PROJECT`
 --------------------------------------
 

docs/data-models.rst

@@ -14,34 +14,44 @@ Project
     :undoc-members:
     :member-order: groupwise
 
+.. _data_models_codebase_resource:
+
 CodebaseResource
 ----------------
 .. autoclass:: scanpipe.models.CodebaseResource()
     :members:
     :undoc-members:
     :member-order: groupwise
 
+.. _data_models_discovered_package:
+
 DiscoveredPackage
 -----------------
 .. autoclass:: scanpipe.models.DiscoveredPackage()
     :members:
     :undoc-members:
     :member-order: groupwise
 
+.. _data_models_discovered_dependency:
+
 DiscoveredDependency
 --------------------
 .. autoclass:: scanpipe.models.DiscoveredDependency()
     :members:
     :undoc-members:
     :member-order: groupwise
 
+.. _data_models_codebase_relation:
+
 CodebaseRelation
 ----------------
 .. autoclass:: scanpipe.models.CodebaseRelation()
     :members:
     :undoc-members:
     :member-order: groupwise
 
+.. _data_models_project_message:
+
 ProjectMessage
 --------------
 .. autoclass:: scanpipe.models.ProjectMessage()

docs/faq.rst

@@ -205,6 +205,21 @@ Also, A new GitHub action is available at
 `scancode-action repository <https://github.com/nexB/scancode-action>`_
 to run ScanCode.io pipelines from your GitHub Workflows.
 
+How can I get notified about my project progression?
+-----------------------------------------------------
+
+You can monitor your project's progress in multiple ways:
+
+- **User Interface:** The project details page provides real-time updates on pipeline
+  execution.
+- **REST API:** Use the API to programmatically check the status of your projects.
+- **CLI Monitoring:** The ``scanpipe list-projects`` command provides an overview of
+  project states.
+- **Webhook Integration:** You can set up webhooks to receive updates in your preferred
+  notification system. For more details, refer to the :ref:`webhooks` section.
+- **Slack notifications:** Get project updates directly in Slack by configuring an
+  incoming webhook. See :ref:`webhooks_slack_notifications` for setup instructions.
+
 .. _faq_tag_input_files:
 
 How to tag input files?
@@ -307,6 +322,39 @@ data older than 7 days::
   command.
 
 How can I provide my license policies?
----------------------------------------
+--------------------------------------
 
 For detailed information about the policies system, refer to :ref:`policies`.
+
+Can you analyze Dockerfiles?
+----------------------------
+
+We have code in https://github.com/aboutcode-org/container-inspector/blob/main/src/container_inspector/dockerfile.py
+for this ... but this may not be wired in other tools at the moment.
+It can for instance map dockerfile instructions to actual docker image history,
+https://github.com/aboutcode-org/container-inspector/blob/main/src/container_inspector/dockerfile.py#L204
+
+Can you analyze a built image? (Build Docker Image Analysis)
+------------------------------------------------------------
+
+Yes, we do this in ScanCode.io. We have one fairly unique feature to actually account
+for all files used in all layers.
+
+Can you analyze all layers of a running container?
+--------------------------------------------------
+
+ScanCode.io scans all layers of images. We can scan all layers of a running container
+if you save the running container as an image first.
+We can also fetch images from registries, local files and technically also from a
+running container, say in a local docker ... but this has not yet been tested so far.
+We do not introspect k8s clusters to analyze the deployed and running images
+there (yet) and that would be a nice future addition.
+For now we can instead work on the many images there, save and analyze them.
+
+Can you analyze Docker in Docker?
+---------------------------------
+
+The input to ScanCode is a local saved image: Docker or OCI.
+Docker in Docker support will demand to have access to the saved images
+(either extracted from the Docker images in Docker, or mounted in a volume or saved
+from the Docker in the Docker image). Once saved we can analyze these alright.