Integration of Clarity compliance mechanism (#1705)

* integrating the clarity-compliance in UI

Signed-off-by: NucleonGodX <[email protected]>

* minor change

Signed-off-by: NucleonGodX <[email protected]>

* have compliance alerts in the single panel

Signed-off-by: NucleonGodX <[email protected]>

* add updated check-compliance

Signed-off-by: NucleonGodX <[email protected]>

* add documentation and support api endpoint for clarity compliance

Signed-off-by: NucleonGodX <[email protected]>

* fix tests

Signed-off-by: NucleonGodX <[email protected]>

* create separate method to retrieve compliance alert

Signed-off-by: NucleonGodX <[email protected]>

* used the new refactored policy code

Signed-off-by: NucleonGodX <[email protected]>

* add test for api and replace clarity_compliance with license_clarity_compliance everywhere

Signed-off-by: NucleonGodX <[email protected]>

* suggestions applied

Signed-off-by: NucleonGodX <[email protected]>

* fix tests

Signed-off-by: NucleonGodX <[email protected]>

---------

Signed-off-by: NucleonGodX <[email protected]>

Author: NucleonGodX

docs/policies.rst

@@ -54,6 +54,43 @@ structure similar to the following:
   - ``warning`` — Use with caution; the license may have some restrictions.
   - ``error`` — The license is prohibited or incompatible with your policy.
 
+Creating Clarity Thresholds Files
+---------------------------------
+
+A valid clarity thresholds file is required to **enable license clarity compliance features**.
+
+The clarity thresholds file, by default named ``policies.yml``, is a **YAML file** with a
+structure similar to the following:
+
+.. code-block:: yaml
+
+    license_clarity_thresholds:
+      91: ok
+      80: warning
+      0: error
+
+- In the example above, the keys ``91``, ``80``, and ``0`` are integer threshold values
+  representing **minimum clarity scores**.
+- The values ``error``, ``warning``, and ``ok`` are the **compliance alert levels** that
+  will be triggered if the project's license clarity score meets or exceeds the
+  corresponding threshold.
+- The thresholds must be listed in **strictly descending order**.
+
+How it works:
+
+- If the clarity score is **91 or above**, the alert is **``ok``**.
+- If the clarity score is **80 to 90**, the alert is **``warning``**.
+- If the clarity score is **below 80**, the alert is **``error``**.
+
+You can adjust the threshold values and alert levels to match your organization's
+compliance requirements.
+
+Accepted values for the alert level:
+
+- ``ok``
+- ``warning``
+- ``error``
+
 App Policies
 ------------
 
@@ -115,7 +152,7 @@ REST API
 --------
 
 For more details on retrieving compliance data through the REST API, see the
-:ref:`rest_api_compliance` section.
+:ref:`rest_api_compliance` section and :ref:`rest_api_clarity_compliance` section.
 
 Command Line Interface
 ----------------------

docs/rest-api.rst

@@ -493,6 +493,31 @@ Data:
         }
     }
 
+.. _rest_api_license_clarity_compliance:
+
+License Clarity Compliance
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This action returns the **license clarity compliance alert** for a project.
+
+The license clarity compliance alert is a single value (``ok``, ``warning``, or ``error``)
+that summarizes the project's **license clarity status**, based on the thresholds defined in
+the ``policies.yml`` file.
+
+``GET /api/projects/6461408c-726c-4b70-aa7a-c9cc9d1c9685/license_clarity_compliance/``
+
+Data:
+    - ``license_clarity_compliance_alert``: The overall license clarity compliance alert
+      for the project.
+
+      Possible values: ``ok``, ``warning``, ``error``.
+
+.. code-block:: json
+
+    {
+        "license_clarity_compliance_alert": "warning"
+    }
+
 Reset
 ^^^^^
 

docs/tutorial_license_policies.rst

@@ -83,6 +83,51 @@ detected license, and computed at the codebase resource level, for example:
       "[...]": "[...]"
     }
 
+License Clarity Thresholds and Compliance
+-----------------------------------------
+
+ScanCode.io also supports **license clarity thresholds**, allowing you to enforce
+minimum standards for license detection quality in your codebase. This is managed
+through the ``license_clarity_thresholds`` section in your ``policies.yml`` file.
+
+Defining Clarity Thresholds
+---------------------------
+
+Add a ``license_clarity_thresholds`` section to your ``policies.yml`` file, for example:
+
+.. code-block:: yaml
+
+    license_clarity_thresholds:
+      91: ok
+      80: warning
+      0: error
+
+
+License Clarity Compliance in Results
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When you run a pipeline with clarity thresholds defined in your ``policies.yml``,
+the computed license clarity compliance alert is included in the project's ``extra_data`` field.
+
+For example:
+
+.. code-block:: json
+
+    "extra_data": {
+      "md5": "d23df4a4",
+      "sha1": "3e9b61cc98c",
+      "size": 3095,
+      "sha256": "abacfc8bcee59067",
+      "sha512": "208f6a83c83a4c770b3c0",
+      "filename": "cuckoo_filter-1.0.6.tar.gz",
+      "sha1_git": "3fdb0f82ad59",
+      "license_clarity_compliance_alert": "error"
+    }
+
+The ``license_clarity_compliance_alert`` value (e.g., ``"error"``, ``"warning"``, or ``"ok"``)
+is computed automatically based on the thresholds you configured and reflects the
+overall license clarity status of the scanned codebase.
+
 Run the ``check-compliance`` command
 ------------------------------------
 
@@ -95,7 +140,7 @@ in the project:
 
 .. code-block:: bash
 
-    4 compliance issues detected on this project.
+    5 compliance issues detected on this project.
     [packages]
      > ERROR: 3
        pkg:pypi/cuckoo-filter@.
@@ -104,6 +149,8 @@ in the project:
     [resources]
      > ERROR: 1
        cuckoo_filter-1.0.6.tar.gz-extract/cuckoo_filter-1.0.6/README.md
+    [license clarity]
+     > ERROR
 
 .. tip::
     In case of compliance alerts, the command returns a non-zero exit code which

scanpipe/api/views.py

@@ -481,6 +481,22 @@ def compliance(self, request, *args, **kwargs):
         compliance_alerts = get_project_compliance_alerts(project, fail_level)
         return Response({"compliance_alerts": compliance_alerts})
 
+    @action(detail=True, methods=["get"])
+    def license_clarity_compliance(self, request, *args, **kwargs):
+        """
+        Retrieve the license clarity compliance alert for a project.
+
+        This endpoint returns the license clarity compliance alert stored in the
+        project's extra_data.
+
+        Example:
+          GET /api/projects/{project_id}/license_clarity_compliance/
+
+        """
+        project = self.get_object()
+        clarity_alert = project.get_license_clarity_compliance_alert()
+        return Response({"license_clarity_compliance_alert": clarity_alert})
+
 
 class RunViewSet(mixins.RetrieveModelMixin, viewsets.GenericViewSet):
     """Add actions to the Run viewset."""

scanpipe/management/commands/check-compliance.py

@@ -74,16 +74,25 @@ def check_compliance(self, fail_level):
             len(issues) for model in alerts.values() for issues in model.values()
         )
 
-        if count and self.verbosity > 0:
-            self.stderr.write(f"{count} compliance issues detected.")
+        clarity_alert = self.project.get_license_clarity_compliance_alert()
+        has_clarity_issue = clarity_alert not in (None, "ok")
+
+        total_issues = count + (1 if has_clarity_issue else 0)
+
+        if total_issues and self.verbosity > 0:
+            self.stderr.write(f"{total_issues} compliance issues detected.")
             for label, model in alerts.items():
                 self.stderr.write(f"[{label}]")
                 for severity, entries in model.items():
                     self.stderr.write(f" > {severity.upper()}: {len(entries)}")
                     if self.verbosity > 1:
                         self.stderr.write("   " + "\n   ".join(entries))
 
-        return count > 0
+            if has_clarity_issue:
+                self.stderr.write("[license clarity]")
+                self.stderr.write(f" > {clarity_alert.upper()}")
+
+        return total_issues > 0
 
     def check_vulnerabilities(self):
         packages = self.project.discoveredpackages.vulnerable_ordered()

scanpipe/models.py

@@ -1544,6 +1544,13 @@ def get_policies_dict(self):
 
         return scanpipe_app.policies
 
+    def get_license_clarity_compliance_alert(self):
+        """
+        Return the license clarity compliance alert value for the project,
+        or None if not set.
+        """
+        return self.extra_data.get("license_clarity_compliance_alert")
+
     def get_license_policy_index(self):
         """Return the policy license index for this project instance."""
         if policies_dict := self.get_policies_dict():

scanpipe/pipes/license_clarity.py

@@ -156,10 +156,29 @@ def load_clarity_thresholds_from_file(file_path):
     file_path = Path(file_path)
 
     if not file_path.exists():
-        return None
+        return
 
     try:
         yaml_content = file_path.read_text(encoding="utf-8")
         return load_clarity_thresholds_from_yaml(yaml_content)
     except (OSError, UnicodeDecodeError) as e:
         raise ValidationError(f"Error reading file {file_path}: {e}")
+
+
+def get_project_clarity_thresholds(project):
+    """
+    Get clarity thresholds for a project using the unified policy loading logic.
+
+    Returns:
+        ClarityThresholdsPolicy or None: Policy object if thresholds are configured
+
+    """
+    policies_dict = project.get_policies_dict()
+    if not policies_dict:
+        return
+
+    clarity_thresholds = policies_dict.get("license_clarity_thresholds")
+    if not clarity_thresholds:
+        return
+
+    return ClarityThresholdsPolicy(clarity_thresholds)

scanpipe/pipes/scancode.py

@@ -60,6 +60,7 @@
 from scanpipe.models import DiscoveredDependency
 from scanpipe.models import DiscoveredPackage
 from scanpipe.pipes import flag
+from scanpipe.pipes.license_clarity import get_project_clarity_thresholds
 
 logger = logging.getLogger("scanpipe.pipes")
 
@@ -1134,7 +1135,10 @@ def make_results_summary(project, scan_results_location):
     Extract selected sections of the Scan results, such as the `summary`
     `license_clarity_score`, and `license_matches` related data.
     The `key_files` are also collected and injected in the `summary` output.
+    Additionally, store license_clarity_compliance_alert in project's extra_data.
     """
+    import json
+
     from scanpipe.api.serializers import CodebaseResourceSerializer
     from scanpipe.api.serializers import DiscoveredPackageSerializer
 
@@ -1167,4 +1171,12 @@ def make_results_summary(project, scan_results_location):
         DiscoveredPackageSerializer(package).data for package in key_files_packages_qs
     ]
 
+    clarity_score = summary.get("license_clarity_score", {}).get("score")
+    if clarity_score is not None:
+        clarity_policy = get_project_clarity_thresholds(project)
+        if clarity_policy:
+            alert = clarity_policy.get_alert_for_score(clarity_score)
+            summary["license_clarity_compliance_alert"] = alert
+
+            project.update_extra_data({"license_clarity_compliance_alert": alert})
     return summary

scanpipe/templates/scanpipe/panels/project_compliance.html

@@ -1,11 +1,11 @@
 {% load humanize %}
-{% if compliance_alerts %}
+{% if compliance_alerts or license_clarity_compliance_alert %}
   <div class="column is-half">
     <nav id="compliance-panel" class="panel is-dark">
       <p class="panel-heading">
         Compliance alerts
       </p>
-      {% for model_name, model_alerts in compliance_alerts.items  %}
+      {% for model_name, model_alerts in compliance_alerts.items %}
         <div class="panel-block">
           <span class="pr-1">
             {{ model_name|title }}
@@ -19,6 +19,20 @@
           {% endfor %}
         </div>
       {% endfor %}
+      {% if license_clarity_compliance_alert %}
+        <div class="panel-block">
+          <span class="pr-1">
+            License clarity
+          </span>
+          <span class="tag is-rounded ml-1
+            {% if license_clarity_compliance_alert == 'error' %}is-danger
+            {% elif license_clarity_compliance_alert == 'warning' %}is-warning
+            {% elif license_clarity_compliance_alert == 'ok' %}is-success
+            {% else %}is-light{% endif %}">
+            {{ license_clarity_compliance_alert|title }}
+          </span>
+        </div>
+      {% endif %}
     </nav>
   </div>
-{% endif %}
+{% endif %}

scanpipe/tests/test_api.py

@@ -1252,6 +1252,24 @@ def test_scanpipe_api_project_action_compliance(self):
         }
         self.assertDictEqual(expected, response.data)
 
+    def test_scanpipe_api_project_action_license_clarity_compliance(self):
+        project = make_project()
+        url = reverse("project-license-clarity-compliance", args=[project.uuid])
+
+        response = self.csrf_client.get(url)
+        expected = {"license_clarity_compliance_alert": None}
+        self.assertEqual(expected, response.data)
+
+        project.update_extra_data({"license_clarity_compliance_alert": "ok"})
+        response = self.csrf_client.get(url)
+        expected = {"license_clarity_compliance_alert": "ok"}
+        self.assertEqual(expected, response.data)
+
+        project.update_extra_data({"license_clarity_compliance_alert": "error"})
+        response = self.csrf_client.get(url)
+        expected = {"license_clarity_compliance_alert": "error"}
+        self.assertEqual(expected, response.data)
+
     def test_scanpipe_api_serializer_get_model_serializer(self):
         self.assertEqual(
             DiscoveredPackageSerializer, get_model_serializer(DiscoveredPackage)