Merge branch 'main' into file-search

Author: aayushkdev

CHANGELOG.rst

@@ -1,6 +1,50 @@
 Changelog
 =========
 
+v35.4.1 (2025-10-24)
+--------------------
+
+- Add ability to download all output results formats as a zipfile for a given project.
+  https://github.com/aboutcode-org/scancode.io/issues/1880
+
+- Add support for tagging inputs in the run management command
+  Add ability to skip the SQLite auto db in combined_run
+  Add documentation to leverage PostgreSQL service
+  https://github.com/aboutcode-org/scancode.io/pull/1916
+
+- Refine d2d pipeline for scala and kotlin.
+  https://github.com/aboutcode-org/scancode.io/issues/1898
+
+- Add utilities to create/init FederatedCode data repo.
+  https://github.com/aboutcode-org/scancode.io/issues/1896
+
+- Add a verify-project CLI management command.
+  https://github.com/aboutcode-org/scancode.io/issues/1903
+
+- Add support for multiple inputs in the run management command.
+  https://github.com/aboutcode-org/scancode.io/issues/1916
+
+- Add the django-htmx app to the stack.
+  https://github.com/aboutcode-org/scancode.io/issues/1917
+
+- Adjust the resource tree view table rendering.
+  https://github.com/aboutcode-org/scancode.io/issues/1840
+
+- Add ".." navigation option in table to navigate to parent resource.
+  https://github.com/aboutcode-org/scancode.io/issues/1869
+
+- Add ability to download all output results formats.
+  https://github.com/aboutcode-org/scancode.io/issues/1880
+
+- Update Java D2D Pipeline to Include Checksum Mapped Sources for Accurate Java Mapping.
+  https://github.com/aboutcode-org/scancode.io/issues/1870
+
+- Auto-detect pipeline from provided input.
+  https://github.com/aboutcode-org/scancode.io/issues/1883
+
+- Migrate SCA workflows verification to new verify-project management command.
+  https://github.com/aboutcode-org/scancode.io/issues/1902
+
 v35.4.0 (2025-09-30)
 --------------------
 

docs/command-line-interface.rst

@@ -419,10 +419,11 @@ Displays status information about the ``PROJECT`` project.
 
 .. _cli_output:
 
-`$ scanpipe output --project PROJECT --format {json,csv,xlsx,spdx,cyclonedx,attribution}`
------------------------------------------------------------------------------------------
+`$ scanpipe output --project PROJECT --format {json,csv,xlsx,spdx,cyclonedx,attribution,...}`
+---------------------------------------------------------------------------------------------
 
-Outputs the ``PROJECT`` results as JSON, XLSX, CSV, SPDX, CycloneDX, and Attribution.
+Outputs the ``PROJECT`` results as JSON, XLSX, CSV, SPDX, CycloneDX,
+ORT package-list.yml, and Attribution.
 The output files are created in the ``PROJECT`` :guilabel:`output/` directory.
 
 Multiple formats can be provided at once::

docs/output-files.rst

@@ -285,7 +285,6 @@ Additional sheets are included **only when relevant** (i.e., when data is availa
 
 SPDX
 ^^^^
-
 ScanCode.io can generate Software Bill of Materials (SBOM) in the **SPDX** format,
 which is an open standard for communicating software component information.
 SPDX is widely used for license compliance, security analysis, and software supply
@@ -309,7 +308,6 @@ The SPDX output includes:
 
 CycloneDX
 ^^^^^^^^^
-
 ScanCode.io can generate **CycloneDX** SBOMs, a lightweight standard designed for
 security and dependency management. CycloneDX is optimized for vulnerability analysis
 and software supply chain risk assessment.

docs/quickstart.rst

@@ -3,8 +3,8 @@
 QuickStart
 ==========
 
-Run a Scan (no installation required!)
---------------------------------------
+Run a Local Directory Scan (no installation required!)
+------------------------------------------------------
 
 The **fastest way** to get started and **scan a codebase** —
 **no installation needed** — is by using the latest
@@ -52,8 +52,120 @@ See the :ref:`RUN command <cli_run>` section for more details on this command.
 .. note::
     Not sure which pipeline to use? Check out :ref:`faq_which_pipeline`.
 
-Next Step: Local Installation
------------------------------
+Run a Remote Package Scan
+-------------------------
+
+Let's look at another example — this time scanning a **remote package archive** by
+providing its **download URL**:
+
+.. code-block:: bash
+
+    docker run --rm \
+      ghcr.io/aboutcode-org/scancode.io:latest \
+      run scan_single_package https://github.com/aboutcode-org/python-inspector/archive/refs/tags/v0.14.4.zip \
+      > results.json
+
+Let's break down what's happening here:
+
+- ``docker run --rm``
+  Runs a temporary container that is automatically removed after the scan completes.
+
+- ``ghcr.io/aboutcode-org/scancode.io:latest``
+  Uses the latest ScanCode.io image from GitHub Container Registry.
+
+- ``run scan_single_package <URL>``
+  Executes the ``scan_single_package`` pipeline, automatically fetching and analyzing
+  the package archive from the provided URL.
+
+- ``> results.json``
+  Writes the scan results to a local ``results.json`` file.
+
+Notice that the ``-v "$(pwd)":/codedrop`` option is **not required** in this case
+because the input is downloaded directly from the provided URL, rather than coming
+from your local filesystem.
+
+The result? A **complete scan of a remote package archive — no setup, one command!**
+
+Use PostgreSQL for Better Performance
+-------------------------------------
+
+By default, ScanCode.io uses a **temporary SQLite database** for simplicity.
+While this works well for quick scans, it has a few limitations — such as
+**no multiprocessing** and slower performance on large codebases.
+
+For improved speed and scalability, you can run your pipelines using a
+**PostgreSQL database** instead.
+
+Start a PostgreSQL Database Service
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+First, start a PostgreSQL container in the background:
+
+.. code-block:: bash
+
+    docker run -d \
+      --name scancodeio-run-db \
+      -e POSTGRES_DB=scancodeio \
+      -e POSTGRES_USER=scancodeio \
+      -e POSTGRES_PASSWORD=scancodeio \
+      -e POSTGRES_INITDB_ARGS="--encoding=UTF-8 --lc-collate=en_US.UTF-8 --lc-ctype=en_US.UTF-8" \
+      -v scancodeio_pgdata:/var/lib/postgresql/data \
+      -p 5432:5432 \
+      postgres:17
+
+This command starts a new PostgreSQL service named ``scancodeio-run-db`` and stores its
+data in a named Docker volume called ``scancodeio_pgdata``.
+
+.. note::
+    You can stop and remove the PostgreSQL service once you are done using:
+
+    .. code-block:: bash
+
+        docker rm -f scancodeio-run-db
+
+.. tip::
+    The named volume ``scancodeio_pgdata`` ensures that your database data
+    **persists across runs**.
+    You can remove it later with ``docker volume rm scancodeio_pgdata`` if needed.
+
+Run a Docker Image Analysis Using PostgreSQL
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Once PostgreSQL is running, you can start a ScanCode.io pipeline
+using the same Docker image, connecting it to the PostgreSQL database container:
+
+.. code-block:: bash
+
+    docker run --rm \
+      --network host \
+      -e SCANCODEIO_NO_AUTO_DB=1 \
+      ghcr.io/aboutcode-org/scancode.io:latest \
+      run analyze_docker_image docker://alpine:3.22.1 \
+      > results.json
+
+Here’s what’s happening:
+
+- ``--network host``
+  Ensures the container can connect to the PostgreSQL service running on your host.
+
+- ``-e SCANCODEIO_NO_AUTO_DB=1``
+  Tells ScanCode.io **not** to create a temporary SQLite database, and instead use
+  the configured PostgreSQL connection defined in its default settings.
+
+- ``ghcr.io/aboutcode-org/scancode.io:latest``
+  Uses the latest ScanCode.io image from GitHub Container Registry.
+
+- ``run analyze_docker_image docker://alpine:3.22.1``
+  Runs the ``analyze_docker_image`` pipeline, scanning the given Docker image.
+
+- ``> results.json``
+  Saves the scan results to a local ``results.json`` file.
+
+The result? A **faster, multiprocessing-enabled scan** backed by PostgreSQL — ideal
+for large or complex analyses.
+
+Next Step: Installation
+-----------------------
 
 Install ScanCode.io, to **unlock all features**:
 

docs/rest-api.rst

@@ -694,10 +694,16 @@ Finally, use this action to download the project results in the provided
 ``output_format`` as an attachment file.
 
 Data:
-    - ``output_format``: ``json``, ``xlsx``, ``spdx``, ``cyclonedx``, ``attribution``
+    - ``output_format``: ``json``, ``xlsx``, ``spdx``, ``cyclonedx``, ``attribution``,
+      ``all_formats``, ``all_outputs``
 
 ``GET /api/projects/d4ed9405-5568-45ad-99f6-782a9b82d1d2/results_download/?output_format=cyclonedx``
 
+.. note::
+   Use ``all_formats`` to generate a zip file containing all output formats for a
+   project, while ``all_outputs`` can be used to obtain a zip file of all existing
+   output files for that project.
+
 .. tip::
   Refer to :ref:`output_files` to learn more about the available output formats.
 

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "scancodeio"
-version = "35.4.0"
+version = "35.4.1"
 description = "Automate software composition analysis pipelines"
 readme = "README.rst"
 requires-python = ">=3.10,<3.14"
@@ -44,6 +44,7 @@ dependencies = [
   "django-filter==25.1",
   "djangorestframework==3.16.1",
   "django-taggit==6.1.0",
+  "django-htmx==1.26.0",
   # Database
   "psycopg[binary]==3.2.10",
   # wait_for_database Django management command

scancodeio/__init__.py

@@ -28,7 +28,7 @@
 
 import git
 
-VERSION = "35.4.0"
+VERSION = "35.4.1"
 
 PROJECT_DIR = Path(__file__).resolve().parent
 ROOT_DIR = PROJECT_DIR.parent
@@ -106,6 +106,9 @@ def combined_run():
     configuration.
     It combines the creation, execution, and result retrieval of the project into a
     single process.
+
+    Set SCANCODEIO_NO_AUTO_DB=1 to use the database configuration from the settings
+    instead of SQLite.
     """
     from django.core.checks.security.base import SECRET_KEY_INSECURE_PREFIX
     from django.core.management import execute_from_command_line
@@ -114,10 +117,12 @@ def combined_run():
     os.environ.setdefault("DJANGO_SETTINGS_MODULE", "scancodeio.settings")
     secret_key = SECRET_KEY_INSECURE_PREFIX + get_random_secret_key()
     os.environ.setdefault("SECRET_KEY", secret_key)
-    os.environ.setdefault("SCANCODEIO_DB_ENGINE", "django.db.backends.sqlite3")
-    os.environ.setdefault("SCANCODEIO_DB_NAME", "scancodeio.sqlite3")
-    # Disable multiprocessing
-    os.environ.setdefault("SCANCODEIO_PROCESSES", "0")
+
+    # Default to SQLite unless SCANCODEIO_NO_AUTO_DB is provided
+    if not os.getenv("SCANCODEIO_NO_AUTO_DB"):
+        os.environ.setdefault("SCANCODEIO_DB_ENGINE", "django.db.backends.sqlite3")
+        os.environ.setdefault("SCANCODEIO_DB_NAME", "scancodeio.sqlite3")
+        os.environ.setdefault("SCANCODEIO_PROCESSES", "0")  # Disable multiprocessing
 
     sys.argv.insert(1, "run")
     execute_from_command_line(sys.argv)

scancodeio/settings.py

@@ -198,6 +198,7 @@
     "django_rq",
     "django_probes",
     "taggit",
+    "django_htmx",
 ]
 
 MIDDLEWARE = [
@@ -208,6 +209,7 @@
     "django.contrib.auth.middleware.AuthenticationMiddleware",
     "django.contrib.messages.middleware.MessageMiddleware",
     "django.middleware.clickjacking.XFrameOptionsMiddleware",
+    "django_htmx.middleware.HtmxMiddleware",
     "scancodeio.middleware.TimezoneMiddleware",
 ]