Merge branch 'main' into 237-custom-pipelines-docs

tdruez · tdruez · commit cdf6d0014764 · 2021-07-20T21:15:08.000+04:00
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
@@ -3,6 +3,10 @@
 
 ### unreleased
 
+- Add ability to register custom pipelines through a new SCANCODEIO_PIPELINES_DIRS
+  setting.
+  https://github.com/nexB/scancode.io/issues/237
+
 - Add a pipeline `scan_package.ScanPackage` to scan a single package archive with
   ScanCode-toolkit.
   https://github.com/nexB/scancode.io/issues/25
diff --git a/docs/introduction.rst b/docs/introduction.rst
@@ -32,16 +32,16 @@ that are adapted to a software project's unique context and technology platform
 can be difficult. This will require deploying and running multiple specialized
 tools and merge their results with a consistent workflow. Moreover,
 when reusing thousands of open source packages is becoming commonplace,
-code scans pipelines need to be scripted as code and running on servers backed
+code scans pipelines need to be scripted as code is running on servers backed
 by a shared database, not on a laptop.
 
 For instance, when you analyze Docker container images, there could be hundreds
 to thousands of system packages, such as Debian, RPM, Alpine, and application
 packages, including npm, PyPI, Rubygems, Maven, installed in an image
-side-by-side with your own code. Taking care of all these can be
+side-by-side with your own code. Taking care of all this can be
 an extremely hard task, and that's when **ScanCode.io** comes into play to help
 organizing these complex code analysis as scripted pipelines and store their
-results in a uniform database for automated code analysis.
+results in a database for automated code analysis.
 
 
 What is ScanPipe?
@@ -73,9 +73,10 @@ are planning to start a new one, consider the following questions:
 2. **Complexity**: Does the project use many third-party components or technologies?
 3. **Reproducibility**: Is it important that the results are reproducible, traceable, and auditable?
 
-If you answered "yes" to any of the above, keep reading - ScanPipe can help you.
-If the answer is "no" to all of the above, which is a valid scenario, e.g., when you
-are doing small-scale analysis, ScanPipe may provide only limited benefit for you.
+If you answered **"yes"** to any of the above, keep reading - ScanPipe can help
+you. If the answer is **"no"** to all of the above, which is a valid scenario,
+e.g., when you are doing small-scale analysis, ScanPipe may provide only limited
+benefit for you.
 
 The first set of available pipelines helps automate the analysis of Docker
 container images and virtual machine (VM) disk images that often harbor
diff --git a/docs/run-docker.rst b/docs/run-docker.rst
@@ -27,8 +27,8 @@ create an environment file, and build the Docker image::
     docker-compose build
 
 .. note::
-    The image will need to be re-build when the ScanCode.io app source code if
-    modified or updated.
+    The image will need to be re-build when the ScanCode.io source code is modified or
+    updated.
 
 Run the Image
 -------------
diff --git a/docs/scancodeio-settings.rst b/docs/scancodeio-settings.rst
@@ -74,6 +74,15 @@ of parallel processes to 4::
 
     SCANCODE_DEFAULT_OPTIONS=--processes 4,--timeout 120
 
+SCANCODEIO_PIPELINES_DIRS
+-------------------------
+
+This setting defines the additional locations ScanCode.io will search for pipelines.
+This should be set to a list of comma-separated strings that contain full paths to your additional
+pipelines directories::
+
+    SCANCODEIO_PIPELINES_DIRS=/var/scancodeio/pipelines/,/home/user/pipelines/
+
 SCANCODEIO_POLICIES_FILE
 ------------------------
 
diff --git a/docs/scanpipe-output.rst b/docs/scanpipe-output.rst
@@ -1,7 +1,7 @@
 .. _scanpipe_output:
 
-Output
-======
+Output Files
+============
 
 Whether you use the command line or the web application to run your
 scans, the generated results are available for review or export in
diff --git a/docs/scanpipe-pipelines.rst b/docs/scanpipe-pipelines.rst
@@ -29,3 +29,8 @@ Scan Codebase
 -------------
 .. autoclass:: scanpipe.pipelines.scan_codebase.ScanCodebase()
     :members:
+
+Scan Package
+------------
+.. autoclass:: scanpipe.pipelines.scan_package.ScanPackage()
+    :members:
diff --git a/docs/tutorial-1.rst b/docs/tutorial-1.rst
@@ -1,7 +1,7 @@
 .. _tutorial_1:
 
-Docker Image Analysis (command line)
-====================================
+Analyze Docker Image (command line)
+===================================
 
 In this tutorial, you will learn by example how to use ScanCode.io to analyze
 a test Docker image by following the given steps and, along the way,
diff --git a/docs/tutorial-2.rst b/docs/tutorial-2.rst
@@ -1,37 +1,69 @@
 .. _tutorial_2:
 
-Scan Codebase (command line)
-============================
+Analyze Codebase (command line)
+===============================
+
+The focus of this tutorial is to guide you through scanning a codebase package
+using ScanCode.io.
+
+.. note::
+    This tutorial assumes you have a current version of ScanCode.io installed
+    locally on your machine. If you do not have it installed,
+    see our :ref:`installation` guide for instructions.
 
 Requirements
 ------------
+Before you follow the instructions in this tutorial, you need to:
 
-- **ScanCode.io is installed**, see :ref:`installation`
-- **Shell access** on the machine where ScanCode.io is installed
+- Install **ScanCode.io** locally
+- Download the following **package archive** and save it to your home directory: `asgiref-3.3.0-py3-none-any.whl <https://files.pythonhosted.org/packages/c0/e8/578887011652048c2d273bf98839a11020891917f3aa638a0bc9ac04d653/asgiref-3.3.0-py3-none-any.whl>`_
+- Have **Shell access** on the machine where ScanCode.io is installed
 
+Instructions
+------------
 
-Before you start
-----------------
+- Open a shell in the ScanCode.io installation directory and activate the
+  virtual environment - **virtualenv**:
 
-Download the following package archive save this in your home directory:
-`asgiref-3.3.0-py3-none-any.whl <https://files.pythonhosted.org/packages/c0/e8/578887011652048c2d273bf98839a11020891917f3aa638a0bc9ac04d653/asgiref-3.3.0-py3-none-any.whl>`_
+.. code-block:: console
 
+    $ source bin/activate
 
-Step-by-step
-------------
+.. code-block:: console
 
-- Open a shell in the ScanCode.io installation directory and activate the virtualenv::
+    >> (scancodeio) $
 
-    $ source bin/activate
+- Create a new project named ``asgiref``:
 
-- The following command will create a new project named ``asgiref``,
-  add the archive as an input for the project,
-  add the ``scan_codebase`` pipeline, and execute it::
+.. code-block:: console
 
-    $ scanpipe create-project asgiref \
-        --input-file ~/asgiref-3.3.0-py3-none-any.whl \
-        --pipeline scan_codebase \
-        --execute
+    $ scanpipe create-project asgiref
+
+.. code-block:: console
+
+    >> Project asgiref created with work directory projects/asgiref-072c89db
+
+- Add the package archive to the project workspace's :guilabel:`input/`
+  directory:
+
+.. code-block:: bash
+
+    $ scanpipe add-input --project asgiref --input-file ~/asgiref-3.3.0-py3-none-any.whl
+
+.. code-block:: console
+
+    >> File(s) copied to the project inputs directory:
+      - asgiref-3.3.0-py3-none-any.whl
+
+- Add the ``scan_codebase`` pipeline to your project:
+
+.. code-block:: console
+
+    $ scanpipe add-pipeline --project asgiref scan_codebase
+
+.. code-block:: console
+
+    >> Pipeline(s) added to the project
 
 .. note::
     The content of the :guilabel:`input/` directory will be copied in the
@@ -41,5 +73,36 @@ Step-by-step
     :guilabel:`codebase/` directory in which case the ``--input`` option can be
     omitted.
 
-- The scan results as JSON and CSV will be available in the project
-  :guilabel:`output/` directory.
+- Run the ``scan_codebase`` pipeline on your project. The pipeline execution
+  progress is shown within the following command's output:
+
+.. code-block:: bash
+
+    $ scanpipe execute --project asgiref
+
+.. code-block:: console
+
+    >> Pipeline scan_codebase run in progress..
+       2021-07-12 17:45:53.85 Pipeline [scan_codebase] starting
+       2021-07-12 17:45:53.85 Step [copy_inputs_to_codebase_directory] starting
+       2021-07-12 17:45:53.86 Step [copy_inputs_to_codebase_directory] completed in 0.00 seconds
+       2021-07-12 17:45:53.86 Step [run_extractcode] starting
+       [...]
+       2021-07-12 17:46:01.61 Pipeline completed
+
+- Finally, you can view your scan results in JSON or CSV file formats inside
+  the project's :guilabel:`output/` directory.
+
+.. tip::
+    The ``inputs`` and ``pipelines`` can be provided at the same time when
+    calling the ``create-project`` command. For instance, the following command
+    will create a new project named ``asgiref``, add the package archive as the
+    project input, add the ``scan_codebase`` pipeline to the project, and
+    execute it:
+
+.. code-block:: bash
+
+    $ scanpipe create-project asgiref \
+        --input-file ~/asgiref-3.3.0-py3-none-any.whl \
+        --pipeline scan_codebase \
+        --execute
diff --git a/docs/user-interface.rst b/docs/user-interface.rst
@@ -90,8 +90,8 @@ to your project by clicking the **"Add inputs"** and **"Add pipeline"** buttons.
 .. image:: images/UI-4-new.png
 
 .. warning::
-    You will not be able to add more inputs once a pipeline has been run on the project.
-    You can add and run more pipelines though.
+    You will not be able to add any extra inputs once a pipeline has been run on
+    the project. However, you still can add and run extra pipelines as needed!
 
 Within each project, you can view your project details, review the results of
 the pipeline execution, or download the output files.
diff --git a/scancodeio/settings/base.py b/scancodeio/settings/base.py
@@ -53,6 +53,11 @@
 
 SCANCODEIO_POLICIES_FILE = env.str("SCANCODEIO_POLICIES_FILE", default="policies.yml")
 
+# This setting defines the additional locations ScanCode.io will search for pipelines.
+# This should be set to a list of strings that contain full paths to your additional
+# pipelines directories.
+SCANCODEIO_PIPELINES_DIRS = env.list("SCANCODEIO_PIPELINES_DIRS", default=[])
+
 # Application definition
 
 INSTALLED_APPS = (
diff --git a/scanpipe/apps.py b/scanpipe/apps.py
@@ -20,6 +20,8 @@
 # ScanCode.io is a free software code scanning tool from nexB Inc. and others.
 # Visit https://github.com/nexB/scancode.io for support and download.
 
+import inspect
+from importlib.machinery import SourceFileLoader
 from pathlib import Path
 
 from django.apps import AppConfig
@@ -55,17 +57,32 @@ def ready(self):
 
     def load_pipelines(self):
         """
-        Load Pipelines from the "scancodeio_pipelines" entry point group.
+        Load Pipelines from the "scancodeio_pipelines" entry point group and from the
+        pipelines Python files found at `SCANCODEIO_PIPELINES_DIRS` locations.
         """
         entry_points = importlib_metadata.entry_points()
 
         # Ignore duplicated entries caused by duplicated paths in `sys.path`.
         pipeline_entry_points = set(entry_points.get("scancodeio_pipelines"))
 
         for entry_point in sorted(pipeline_entry_points):
-            pipeline_class = entry_point.load()
-            pipeline_name = entry_point.name
-            self.register_pipeline(pipeline_name, pipeline_class)
+            self.register_pipeline(name=entry_point.name, cls=entry_point.load())
+
+        # Register user provided pipelines
+        pipelines_dirs = getattr(settings, "SCANCODEIO_PIPELINES_DIRS", [])
+
+        for pipelines_dir in pipelines_dirs:
+            pipelines_path = Path(pipelines_dir)
+
+            if not pipelines_path.is_dir():
+                raise ImproperlyConfigured(
+                    f'The provided pipelines directory "{pipelines_dir}" in '
+                    f"the SCANCODEIO_PIPELINES_DIRS setting is not available."
+                )
+
+            python_files = pipelines_path.rglob("*.py")
+            for path in python_files:
+                self.register_pipeline_from_file(path)
 
     def register_pipeline(self, name, cls):
         """
@@ -83,6 +100,28 @@ def register_pipeline(self, name, cls):
 
         self._pipelines[name] = cls
 
+    def register_pipeline_from_file(self, path):
+        """
+        Search for a Pipeline subclass in the provided file `path` and register it
+        when found.
+        """
+        module_name = inspect.getmodulename(path)
+        module = SourceFileLoader(module_name, str(path)).load_module()
+
+        def is_local_module_pipeline(obj):
+            return is_pipeline(obj) and obj.__module__ == module_name
+
+        pipeline_classes = inspect.getmembers(module, is_local_module_pipeline)
+
+        if len(pipeline_classes) > 1:
+            raise ImproperlyConfigured(
+                f"Only one Pipeline class allowed per pipeline file: {path}."
+            )
+
+        elif pipeline_classes:
+            pipeline_class = pipeline_classes[0][1]
+            self.register_pipeline(name=module_name, cls=pipeline_class)
+
     @property
     def pipelines(self):
         return dict(self._pipelines)
diff --git a/scanpipe/pipelines/__init__.py b/scanpipe/pipelines/__init__.py
@@ -63,8 +63,8 @@ def get_steps(cls):
             return cls.steps()
 
         warnings.warn(
-            "Defining ``steps`` as a tuple is deprecated. "
-            "Use a ``steps(cls)`` classmethod instead."
+            f"Defining ``steps`` as a tuple is deprecated in {cls} "
+            f"Use a ``steps(cls)`` classmethod instead."
         )
         return cls.steps
 
diff --git a/scanpipe/tests/pipelines/register_from_file.py b/scanpipe/tests/pipelines/register_from_file.py
@@ -0,0 +1,36 @@
+# SPDX-License-Identifier: Apache-2.0
+#
+# http://nexb.com and https://github.com/nexB/scancode.io
+# The ScanCode.io software is licensed under the Apache License version 2.0.
+# Data generated with ScanCode.io is provided as-is without warranties.
+# ScanCode is a trademark of nexB Inc.
+#
+# You may not use this software except in compliance with the License.
+# You may obtain a copy of the License at: http://apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software distributed
+# under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
+# CONDITIONS OF ANY KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations under the License.
+#
+# Data Generated with ScanCode.io is provided on an "AS IS" BASIS, WITHOUT WARRANTIES
+# OR CONDITIONS OF ANY KIND, either express or implied. No content created from
+# ScanCode.io should be considered or used as legal advice. Consult an Attorney
+# for any legal advice.
+#
+# ScanCode.io is a free software code scanning tool from nexB Inc. and others.
+# Visit https://github.com/nexB/scancode.io for support and download.
+
+from scanpipe.tests.pipelines.do_nothing import DoNothing
+
+
+class RegisterFromFile(DoNothing):
+    """
+    A pipeline to be registered from its file path.
+    """
+
+    @classmethod
+    def steps(cls):
+        return (cls.step1,)
+
+    def step1(self):
+        pass
diff --git a/scanpipe/tests/test_apps.py b/scanpipe/tests/test_apps.py
diff --git a/scanpipe/tests/test_pipelines.py b/scanpipe/tests/test_pipelines.py
