Merge remote-tracking branch 'upstream/main' into clarity-compliance-integration

Author: NucleonGodX

CHANGELOG.rst

@@ -1,9 +1,35 @@
 Changelog
 =========
 
-v35.1.0 (unreleased)
+v35.2.0 (unreleased)
 --------------------
 
+- Refactor policies implementation to support more than licenses.
+  The entire ``policies`` data is now stored on the ``ScanPipeConfig`` in place of the
+  ``license_policy_index``.
+  Also, a new method ``get_policies_dict`` methods is now available on the ``Project``
+  model to easily retrieve all the policies data as a dictionary.
+  Renamed for clarity:
+  * ``policy_index`` to ``license_policy_index``
+  * ``policies_enabled`` to ``license_policies_enabled``
+  https://github.com/aboutcode-org/scancode.io/pull/1718
+
+- Add support for SPDX license identifiers as ``license_key`` in license policies
+  ``policies.yml`` file.
+  https://github.com/aboutcode-org/scancode.io/issues/1348
+
+v35.1.0 (2025-07-02)
+--------------------
+
+- Replace the ``setup.py``/``setup.cfg`` by ``pyproject.toml`` file.
+  https://github.com/aboutcode-org/scancode.io/issues/1608
+
+- Update scancode-toolkit to v32.4.0. See CHANGELOG for updates:
+  https://github.com/aboutcode-org/scancode-toolkit/releases/tag/v32.4.0
+  Adds a new ``git_sha1`` attribute to the ``CodebaseResource`` model as this
+  is now computed and returned from the ``scancode-toolkit`` ``--info`` plugin.
+  https://github.com/aboutcode-org/scancode.io/pull/1708
+
 - Add a ``--fail-on-vulnerabilities`` option in ``check-compliance`` management command.
   When this option is enabled, the command will exit with a non-zero status if known
   vulnerabilities are detected in discovered packages and dependencies.
@@ -16,6 +42,10 @@ v35.1.0 (unreleased)
   license rules used during the scan.
   https://github.com/aboutcode-org/scancode.io/issues/1657
 
+- Add a new step to the ``DeployToDevelop`` pipeline, ``map_python``, to match
+  Cython source files (.pyx) to their compiled binaries.
+  https://github.com/aboutcode-org/scancode.io/pull/1703
+
 v35.0.0 (2025-06-23)
 --------------------
 

Dockerfile

@@ -87,7 +87,7 @@ RUN mkdir -p /var/$APP_NAME/static/ \
  && mkdir -p /var/$APP_NAME/workspace/
 
 # Install the dependencies before the codebase COPY for proper Docker layer caching
-COPY --chown=$APP_USER:$APP_USER setup.cfg setup.py $APP_DIR/
+COPY --chown=$APP_USER:$APP_USER pyproject.toml $APP_DIR/
 RUN pip install --no-cache-dir .
 
 # Copy the codebase and set the proper permissions for the APP_USER

Makefile

@@ -143,10 +143,6 @@ docs:
 	rm -rf docs/_build/
 	@${ACTIVATE} sphinx-build docs/ docs/_build/
 
-bump:
-	@echo "-> Bump the version"
-	@${ACTIVATE} bumpver update --no-fetch --patch
-
 docker-images:
 	@echo "-> Build Docker services"
 	docker compose build
@@ -163,4 +159,4 @@ offline-package: docker-images
 	@mkdir -p dist/
 	@tar -cf dist/scancodeio-offline-package-`git describe --tags`.tar build/
 
-.PHONY: virtualenv conf dev envfile install doc8 check valid check-deploy clean migrate upgrade postgresdb sqlitedb backupdb run run-docker-dev test fasttest docs bump docker-images offline-package
+.PHONY: virtualenv conf dev envfile install doc8 check valid check-deploy clean migrate upgrade postgresdb sqlitedb backupdb run run-docker-dev test fasttest docs docker-images offline-package

RELEASE.md

@@ -4,7 +4,7 @@
 
 - Create a new `release-x.x.x` branch
 - Update the version in:
-  - `setup.cfg` (2 entries)
+  - `pyproject.toml`
   - `scancodeio/__init__.py`
   - `CHANGELOG.rst` (set date)
 - Commit and push this branch
@@ -24,7 +24,7 @@
 ```
 cd scancode.io
 source .venv/bin/activate
-pip install build
+python -m pip install build
 python -m build --sdist --wheel --outdir dist/ .
 ```
 

docs/introduction.rst

@@ -91,8 +91,8 @@ The **Django framework** is leveraged for many aspects of ScanCode.io:
 
 .. note::
     Multiple applications from the Django eco-system are also included,
-    see the `setup.cfg <https://github.com/aboutcode-org/scancode.io/blob/main/setup.cfg>`_ file
-    for an exhaustive list of dependencies.
+    see the `pyproject.toml <https://github.com/aboutcode-org/scancode.io/blob/main/pyproject.toml>`_
+    file for an exhaustive list of dependencies.
 
 The second essential part of ScanCode.io is the **ScanCode Toolkit**, which is used
 for archives extraction and as the scanning engine.

docs/policies.rst

@@ -20,23 +20,39 @@ structure similar to the following:
       - license_key: mit
         label: Approved License
         compliance_alert: ''
+
       - license_key: mpl-2.0
         label: Restricted License
         compliance_alert: warning
+
       - license_key: gpl-3.0
         label: Prohibited License
         compliance_alert: error
 
-- In the example above, licenses are referenced by the ``license_key``,
-  such as `mit` and `gpl-3.0`, which represent the ScanCode license keys used to
-  match against licenses detected in scan results.
-- Each policy is defined with a ``label`` and a ``compliance_alert``.
-  You can customize the labels as desired.
-- The ``compliance_alert`` field accepts three values:
+      - license_key: OFL-1.1
+        compliance_alert: warning
+
+      - license_key: LicenseRef-scancode-public-domain
+        compliance_alert: ''
+
+      - license_key: LicenseRef-scancode-unknown-license-reference
+        compliance_alert: error
+
+- In the example above, licenses are referenced using the ``license_key`` field.
+  These keys can be either **ScanCode license identifiers** (e.g., "mit", "gpl-3.0"),
+  or **SPDX license identifiers** (e.g., "OFL-1.1",
+  "LicenseRef-scancode-public-domain").
+  These values are used to match against the licenses detected in scan results.
+
+- Each policy entry includes a ``label`` and a ``compliance_alert`` field.
+  The ``label`` is a customizable description used for display or reporting purposes.
+
+- The ``compliance_alert`` field determines the severity level for a license and
+  supports the following values:
 
-   - ``''`` (empty string)
-   - ``warning``
-   - ``error``
+  - ``''`` (empty string) — No action needed; the license is approved.
+  - ``warning`` — Use with caution; the license may have some restrictions.
+  - ``error`` — The license is prohibited or incompatible with your policy.
 
 Creating Clarity Thresholds Files
 ---------------------------------

docs/rest-api.rst

@@ -556,7 +556,7 @@ File content
 This displays the content of a ``project`` file resource provided using the
 ``?path=<resource_path>`` argument.
 
-``GET /api/projects/d4ed9405-5568-45ad-99f6-782a9b82d1d2/file_content/?path=setup.py``
+``GET /api/projects/d4ed9405-5568-45ad-99f6-782a9b82d1d2/file_content/?path=filename.ext``
 
 .. code-block:: json
 

pyproject.toml

@@ -1,3 +1,156 @@
+[build-system]
+requires = ["setuptools"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "scancodeio"
+version = "35.1.0"
+description = "Automate software composition analysis pipelines"
+readme = "README.rst"
+requires-python = ">=3.10,<3.14"
+license = "Apache-2.0"
+license-files = ["LICENSE", "NOTICE", "scan.NOTICE"]
+authors = [
+    { name = "nexB Inc.", email = "[email protected]" }
+]
+keywords = [
+    "open source", "scan", "license", "package", "dependency",
+    "copyright", "filetype", "author", "extract", "licensing",
+    "scancode", "scanpipe", "docker", "rootfs", "vm",
+    "virtual machine", "pipeline", "code analysis", "container"
+]
+classifiers = [
+    "Development Status :: 5 - Production/Stable",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Information Technology",
+    "Intended Audience :: Legal Industry",
+    "Programming Language :: Python",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Utilities"
+]
+dependencies = [
+  "importlib-metadata==8.7.0",
+  "setuptools==80.9.0",
+  # Django related
+  "Django==5.1.11",
+  "django-environ==0.12.0",
+  "django-crispy-forms==2.4",
+  "crispy-bootstrap3==2024.1",
+  "django-filter==25.1",
+  "djangorestframework==3.16.0",
+  "django-taggit==6.1.0",
+  # Database
+  "psycopg[binary]==3.2.9",
+  # wait_for_database Django management command
+  "django-probes==1.7.0",
+  # Task queue
+  "rq==2.4.0",
+  "django-rq==3.0.1",
+  "redis==6.2.0",
+  # WSGI server
+  "gunicorn==23.0.0",
+  # Docker
+  "container-inspector==33.0.0",
+  # ScanCode-toolkit
+  "scancode-toolkit[packages]==32.4.0",
+  "extractcode[full]==31.0.0",
+  "commoncode==32.3.0",
+  "Beautifulsoup4[chardet]==4.13.4",
+  "packageurl-python==0.17.1",
+  # FetchCode
+  "fetchcode-container==1.2.3.210512; sys_platform == 'linux'",
+  # Inspectors
+  "elf-inspector==0.0.3",
+  "go-inspector==0.5.0",
+  "rust-inspector==0.1.0",
+  "binary-inspector==0.1.2",
+  "python-inspector==0.14.0",
+  "source-inspector==0.7.0; sys_platform != 'darwin' and platform_machine != 'arm64'",
+  "aboutcode-toolkit==11.1.1",
+  # Utilities
+  "XlsxWriter==3.2.5",
+  "openpyxl==3.1.5",
+  "requests==2.32.4",
+  "GitPython==3.1.44",
+  # Profiling
+  "pyinstrument==5.0.2",
+  # CycloneDX
+  "cyclonedx-python-lib==10.2.0",
+  "jsonschema==4.24.0",
+  # MatchCode-toolkit
+  "matchcode-toolkit==7.2.2",
+  # Univers
+  "univers==31.0.0",
+  # Markdown
+  "markdown-it-py==3.0.0",
+  "bleach==6.2.0",
+  # Antivirus
+  "clamd==1.0.2",
+  # FederatedCode
+  "aboutcode.hashid==0.2.0",
+  # AboutCode pipeline
+  "aboutcode.pipeline==0.2.1",
+  "scipy==1.15.3"
+]
+
+[project.optional-dependencies]
+dev = [
+  # Validation
+  "ruff==0.12.0",
+  "doc8==2.0.0",
+  # Debug
+  "django-debug-toolbar==5.2.0",
+  # Documentation
+  "Sphinx==8.1.3",
+  "sphinx-rtd-theme==3.0.2",
+  "sphinx-rtd-dark-mode==1.3.0",
+  "sphinxcontrib-django==2.5",
+]
+android_analysis = [
+  "android_inspector==0.0.1"
+]
+
+[project.urls]
+Homepage = "https://github.com/aboutcode-org/scancode.io"
+Documentation = "https://scancodeio.readthedocs.io/"
+Repository = "https://github.com/aboutcode-org/scancode.io.git"
+Issues = "https://github.com/aboutcode-org/scancode.io/issues"
+Changelog = "https://github.com/aboutcode-org/scancode.io/blob/main/CHANGELOG.rst"
+
+[project.scripts]
+scanpipe = "scancodeio:command_line"
+run = "scancodeio:combined_run"
+
+[project.entry-points."scancodeio_pipelines"]
+analyze_docker_image = "scanpipe.pipelines.analyze_docker:Docker"
+analyze_root_filesystem_or_vm_image = "scanpipe.pipelines.analyze_root_filesystem:RootFS"
+analyze_windows_docker_image = "scanpipe.pipelines.analyze_docker_windows:DockerWindows"
+collect_strings_gettext = "scanpipe.pipelines.collect_strings_gettext:CollectStringsGettext"
+collect_symbols_ctags = "scanpipe.pipelines.collect_symbols_ctags:CollectSymbolsCtags"
+collect_symbols_pygments = "scanpipe.pipelines.collect_symbols_pygments:CollectSymbolsPygments"
+collect_symbols_tree_sitter = "scanpipe.pipelines.collect_symbols_tree_sitter:CollectSymbolsTreeSitter"
+enrich_with_purldb = "scanpipe.pipelines.enrich_with_purldb:EnrichWithPurlDB"
+find_vulnerabilities = "scanpipe.pipelines.find_vulnerabilities:FindVulnerabilities"
+inspect_elf_binaries = "scanpipe.pipelines.inspect_elf_binaries:InspectELFBinaries"
+inspect_packages = "scanpipe.pipelines.inspect_packages:InspectPackages"
+load_inventory = "scanpipe.pipelines.load_inventory:LoadInventory"
+load_sbom = "scanpipe.pipelines.load_sbom:LoadSBOM"
+map_deploy_to_develop = "scanpipe.pipelines.deploy_to_develop:DeployToDevelop"
+match_to_matchcode = "scanpipe.pipelines.match_to_matchcode:MatchToMatchCode"
+populate_purldb = "scanpipe.pipelines.populate_purldb:PopulatePurlDB"
+publish_to_federatedcode = "scanpipe.pipelines.publish_to_federatedcode:PublishToFederatedCode"
+resolve_dependencies = "scanpipe.pipelines.resolve_dependencies:ResolveDependencies"
+scan_codebase = "scanpipe.pipelines.scan_codebase:ScanCodebase"
+scan_for_virus = "scanpipe.pipelines.scan_for_virus:ScanForVirus"
+scan_single_package = "scanpipe.pipelines.scan_single_package:ScanSinglePackage"
+
+[tool.setuptools.packages.find]
+where = ["."]
+
 [tool.ruff]
 line-length = 88
 extend-exclude = ["migrations", "var"]
@@ -11,9 +164,12 @@ select = [
     "D",  # pydocstyle
     "F",  # Pyflakes
     "UP", # pyupgrade
+    "DJ", # flake8-django
     "S",  # flake8-bandit
     "I",  # isort
     "C9", # McCabe complexity
+    "FIX", # flake8-fix
+    "FURB", # refurb
 ]
 ignore = ["D1", "D203", "D205", "D212", "D400", "D415"]
 
@@ -35,6 +191,5 @@ max-complexity = 10
 [tool.ruff.lint.per-file-ignores]
 # Allow the usage of assert in the test_spdx file.
 "**/test_spdx.py*" = ["S101"]
-"scanpipe/pipes/spdx.py" = ["UP006", "UP035"]
 # Allow complexity in management commands
 "scanpipe/management/commands/*" = ["C901"]

scancodeio/__init__.py

@@ -28,7 +28,7 @@
 
 import git
 
-VERSION = "35.0.0"
+VERSION = "35.1.0"
 
 PROJECT_DIR = Path(__file__).resolve().parent
 ROOT_DIR = PROJECT_DIR.parent

scancodeio/settings.py

@@ -292,6 +292,7 @@
     SCANCODEIO_WORKSPACE_LOCATION = tempfile.mkdtemp()
     SCANCODEIO_REQUIRE_AUTHENTICATION = True
     SCANCODEIO_SCAN_FILE_TIMEOUT = 120
+    SCANCODEIO_POLICIES_FILE = None
     # The default password hasher is rather slow by design.
     # Using a faster hashing algorithm in the testing context to speed up the run.
     PASSWORD_HASHERS = ["django.contrib.auth.hashers.MD5PasswordHasher"]