Skip to content

Commit 881e62d

Browse files
tdruez404-geek
andauthored
Scorecard integration internal (#1777)
* Add ScoreCard config into settings.py developed functions to check for availability nexB#598 Signed-off-by: 404-geek <[email protected]> * code style fix nexB#598 Signed-off-by: 404-geek <[email protected]> * settings.py code style fix nexB#598 Signed-off-by: 404-geek <[email protected]> * mixin import and models declaration nexB#1283 Signed-off-by: 404-geek <[email protected]> * transforming scorecard data into object for saving nexB#1283 Signed-off-by: 404-geek <[email protected]> * added test cases for saving scorecard data into models and modified saving logic nexB#1283 Signed-off-by: 404-geek <[email protected]> * added score checks mixin to models nexB#1283 Signed-off-by: 404-geek <[email protected]> * empty details in score response handled nexB#1283 Signed-off-by: 404-geek <[email protected]> * changed class names to camel case for models and modified the tests nexB#1283 Signed-off-by: 404-geek <[email protected]> * code formatted nexB#1283 Signed-off-by: 404-geek <[email protected]> * code formatted nexB#1283 Signed-off-by: 404-geek <[email protected]> * code formatted nexB#1283 Signed-off-by: 404-geek <[email protected]> * docstrings formatted nexB#1283 Signed-off-by: 404-geek <[email protected]> * created basic fetch and availability functions for scorecode pipeline nexB#598 Signed-off-by: 404-geek <[email protected]> * modified doc strings and models and imported ScoreCode package in setup.cfg nexB#1283 Signed-off-by: 404-geek <[email protected]> * setup.cfg nexB#1283 Signed-off-by: 404-geek <[email protected]> * reinstated deleted code during rebase nexB#1283 Signed-off-by: 404-geek <[email protected]> * code formatting nexB#1283 Signed-off-by: 404-geek <[email protected]> * database migrations for scorecard nexB#1283 Signed-off-by: 404-geek <[email protected]> * updated the scanpipe only fields nexB#1283 Signed-off-by: 404-geek <[email protected]> * changed scorecode commit hash for latest pull nexB#1283 Signed-off-by: 404-geek <[email protected]> * update pipeline code and changed scorecode hash commit nexB#1283 Signed-off-by: 404-geek <[email protected]> * changed imports structure nexB#1283 Signed-off-by: 404-geek <[email protected]> * modified lookup and save logic nexB#1283 Signed-off-by: 404-geek <[email protected]> * merged migrations due to conflicts nexB#1283 Signed-off-by: 404-geek <[email protected]> * updated migrations nexB#1283 Signed-off-by: 404-geek <[email protected]> * updated doc string for get_scorecard_info_packages.py nexB#1283 Signed-off-by: 404-geek <[email protected]> * Added scorecard pipeline to SCIO with intergration test nexB#1283 Signed-off-by: 404-geek <[email protected]> * moved the data to be regenerated if reqiured nexB#1283 Signed-off-by: 404-geek <[email protected]> * updated urls for testing nexB#1283 Signed-off-by: 404-geek <[email protected]> * added merged migration file nexB#1283 Signed-off-by: 404-geek <[email protected]> * Changed docstring and renamed functions according to suggestions nexB#1283 Signed-off-by: 404-geek <[email protected]> * class name changes in steps of pipeline nexB#1283 Signed-off-by: 404-geek <[email protected]> * pipeline name updated nexB#1283 Signed-off-by: 404-geek <[email protected]> * update pipeline code and steps nexB#1283 Signed-off-by: 404-geek <[email protected]> * update pipeline steps to work with scorecode 0.0.2 release nexB#1283 Signed-off-by: 404-geek <[email protected]> * update migration nexB#1283 Signed-off-by: 404-geek <[email protected]> * update migration nexB#1283 Signed-off-by: 404-geek <[email protected]> * rename pipeline name with data parsing function nexB#1283 Signed-off-by: 404-geek <[email protected]> * code valid nexB#1283 Signed-off-by: 404-geek <[email protected]> * update setup.cfg nexB#1283 Signed-off-by: 404-geek <[email protected]> * optimize code while saving score checks nexB#1283 Signed-off-by: 404-geek <[email protected]> * update test cases and regen scorecard data logic nexB#1283 Signed-off-by: 404-geek <[email protected]> * update migration file nexB#1283 Signed-off-by: 404-geek <[email protected]> * remove unwanted change nexB#1283 Signed-off-by: 404-geek <[email protected]> * unit tests for scorecard model functions and minor fixes nexB#1283 Signed-off-by: 404-geek <[email protected]> * updated built-in-pipelines.rst with `ScoreCode` pipeline Signed-off-by: 404-geek <[email protected]> * fix path bugs Signed-off-by: 404-geek <[email protected]> * change migration file and update `ScoreCode` version Signed-off-by: 404-geek <[email protected]> * cosmetic changes to pipeline Signed-off-by: 404-geek <[email protected]> * Refactor Code Signed-off-by: 404-geek <[email protected]> * Refactor Code Signed-off-by: 404-geek <[email protected]> * Migration Script Signed-off-by: 404-geek <[email protected]> * Update Migration and redundant check of scores Signed-off-by: 404-geek <[email protected]> * valid check Signed-off-by: 404-geek <[email protected]> * Regen the migration file for proper ordering Signed-off-by: tdruez <[email protected]> * Refine the naming and documentation Signed-off-by: tdruez <[email protected]> * Improve the pipeline integration unit test Signed-off-by: tdruez <[email protected]> * Add changelog entry Signed-off-by: tdruez <[email protected]> --------- Signed-off-by: 404-geek <[email protected]> Signed-off-by: tdruez <[email protected]> Co-authored-by: 404-geek <[email protected]>
1 parent 70f6da3 commit 881e62d

File tree

11 files changed

+1261
-10
lines changed

11 files changed

+1261
-10
lines changed

CHANGELOG.rst

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,11 @@ v35.2.0 (unreleased)
2222
Vulnerabilities and compliance alert are displayed along the dependency entries.
2323
https://github.com/aboutcode-org/scancode.io/pull/1742
2424

25+
- Add new ``fetch_scores`` pipeline.
26+
This pipeline retrieves ScoreCode data for each discovered package in the project
27+
and stores it in the corresponding package instances.
28+
https://github.com/aboutcode-org/scancode.io/pull/1294
29+
2530
v35.1.0 (2025-07-02)
2631
--------------------
2732

docs/built-in-pipelines.rst

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,6 @@ Collect symbols and string with Tree-Sitter (addon)
8282

8383
Enrich With PurlDB (addon)
8484
--------------------------
85-
8685
.. warning::
8786
This pipeline requires access to a PurlDB service.
8887
Refer to :ref:`scancodeio_settings_purldb` to configure access to PurlDB in your
@@ -96,7 +95,6 @@ Enrich With PurlDB (addon)
9695

9796
Find Vulnerabilities (addon)
9897
----------------------------
99-
10098
.. warning::
10199
This pipeline requires access to a VulnerableCode database.
102100
Refer to :ref:`scancodeio_settings_vulnerablecode` to configure access to
@@ -150,7 +148,6 @@ Resolve Dependencies
150148

151149
Map Deploy To Develop
152150
---------------------
153-
154151
.. warning::
155152
This pipeline requires input files to be tagged with the following:
156153

@@ -168,7 +165,6 @@ Map Deploy To Develop
168165

169166
Match to MatchCode (addon)
170167
--------------------------
171-
172168
.. warning::
173169
This pipeline requires access to a MatchCode.io service.
174170
Refer to :ref:`scancodeio_settings_matchcodeio` to configure access to
@@ -182,7 +178,6 @@ Match to MatchCode (addon)
182178

183179
Populate PurlDB (addon)
184180
-----------------------
185-
186181
.. warning::
187182
This pipeline requires access to a PurlDB service.
188183
Refer to :ref:`scancodeio_settings_purldb` to configure access to PurlDB in your
@@ -196,7 +191,6 @@ Populate PurlDB (addon)
196191

197192
Publish To FederatedCode (addon)
198193
--------------------------------
199-
200194
.. warning::
201195
This pipeline requires access to a FederatedCode service.
202196
Refer to :ref:`scancodeio_settings_federatedcode` to configure access to
@@ -229,3 +223,13 @@ Scan Single Package
229223
.. autoclass:: scanpipe.pipelines.scan_single_package.ScanSinglePackage()
230224
:members:
231225
:member-order: bysource
226+
227+
Fetch Scores (addon)
228+
--------------------
229+
.. warning::
230+
This pipeline is preconfigured to access the "OpenSSF Scorecard API"
231+
available at https://api.securityscorecards.dev/
232+
233+
.. autoclass:: scanpipe.pipelines.fetch_scores.FetchScores()
234+
:members:
235+
:member-order: bysource

docs/policies.rst

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -152,7 +152,8 @@ REST API
152152
--------
153153

154154
For more details on retrieving compliance data through the REST API, see the
155-
:ref:`rest_api_compliance` section and :ref:`rest_api_clarity_compliance` section.
155+
:ref:`rest_api_compliance` section and :ref:`rest_api_license_clarity_compliance`
156+
section.
156157

157158
Command Line Interface
158159
----------------------

pyproject.toml

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -94,7 +94,10 @@ dependencies = [
9494
"aboutcode.hashid==0.2.0",
9595
# AboutCode pipeline
9696
"aboutcode.pipeline==0.2.1",
97-
"scipy==1.15.3"
97+
"scipy==1.15.3",
98+
# ScoreCode
99+
"scorecode==0.0.4",
100+
98101
]
99102

100103
[project.optional-dependencies]
@@ -134,6 +137,7 @@ collect_symbols_ctags = "scanpipe.pipelines.collect_symbols_ctags:CollectSymbols
134137
collect_symbols_pygments = "scanpipe.pipelines.collect_symbols_pygments:CollectSymbolsPygments"
135138
collect_symbols_tree_sitter = "scanpipe.pipelines.collect_symbols_tree_sitter:CollectSymbolsTreeSitter"
136139
enrich_with_purldb = "scanpipe.pipelines.enrich_with_purldb:EnrichWithPurlDB"
140+
fetch_scores = "scanpipe.pipelines.fetch_scores:FetchScores"
137141
find_vulnerabilities = "scanpipe.pipelines.find_vulnerabilities:FindVulnerabilities"
138142
inspect_elf_binaries = "scanpipe.pipelines.inspect_elf_binaries:InspectELFBinaries"
139143
inspect_packages = "scanpipe.pipelines.inspect_packages:InspectPackages"
Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# Generated by Django 5.1.11 on 2025-07-25 12:55
2+
3+
import django.db.models.deletion
4+
import uuid
5+
from django.db import migrations, models
6+
7+
8+
class Migration(migrations.Migration):
9+
10+
dependencies = [
11+
('scanpipe', '0075_codebaseresource_parent_path_and_more'),
12+
]
13+
14+
operations = [
15+
migrations.CreateModel(
16+
name='DiscoveredPackageScore',
17+
fields=[
18+
('scoring_tool', models.CharField(blank=True, choices=[('ossf-scorecard', 'Ossf'), ('others', 'Others')], help_text='Defines the source of a score or any other scoring metricsFor example: ossf-scorecard for scorecard data', max_length=100)),
19+
('scoring_tool_version', models.CharField(blank=True, help_text='Defines the version of the scoring tool used for scanning thepackageFor Eg : 4.6 current version of OSSF - scorecard', max_length=50)),
20+
('score', models.CharField(blank=True, help_text='Score of the package which is scanned', max_length=50)),
21+
('scoring_tool_documentation_url', models.CharField(blank=True, help_text='Documentation URL of the scoring tool used', max_length=100)),
22+
('score_date', models.DateTimeField(blank=True, editable=False, help_text='Date when the scoring was calculated on the package', null=True)),
23+
('uuid', models.UUIDField(db_index=True, default=uuid.uuid4, editable=False, primary_key=True, serialize=False, verbose_name='UUID')),
24+
('discovered_package', models.ForeignKey(editable=False, help_text='The package for which the score is given', on_delete=django.db.models.deletion.CASCADE, related_name='scores', to='scanpipe.discoveredpackage')),
25+
],
26+
options={
27+
'verbose_name': 'discovered package score',
28+
'verbose_name_plural': 'discovered package scores',
29+
'ordering': ['-score'],
30+
},
31+
),
32+
migrations.CreateModel(
33+
name='ScorecardCheck',
34+
fields=[
35+
('check_name', models.CharField(blank=True, help_text='Defines the name of check corresponding to the OSSF scoreFor example: Code-Review or CII-Best-PracticesThese are the some of the checks which are performed on a scanned package', max_length=100)),
36+
('check_score', models.CharField(blank=True, help_text='Defines the score of the check for the package scannedFor Eg : 9 is a score given for Code-Review', max_length=50)),
37+
('reason', models.CharField(blank=True, help_text='Gives a reason why a score was given for a specific checkFor eg, : Found 9/10 approved changesets -- score normalized to 9', max_length=300)),
38+
('details', models.JSONField(blank=True, default=list, help_text='A list of details/errors regarding the score')),
39+
('uuid', models.UUIDField(db_index=True, default=uuid.uuid4, editable=False, primary_key=True, serialize=False, verbose_name='UUID')),
40+
('package_score', models.ForeignKey(editable=False, help_text='The checks for which the score is given', on_delete=django.db.models.deletion.CASCADE, related_name='checks', to='scanpipe.discoveredpackagescore')),
41+
],
42+
options={
43+
'verbose_name': 'scorecard check',
44+
'verbose_name_plural': 'scorecard checks',
45+
'ordering': ['-check_score'],
46+
},
47+
),
48+
migrations.AddIndex(
49+
model_name='discoveredpackagescore',
50+
index=models.Index(fields=['score'], name='scanpipe_di_score_078964_idx'),
51+
),
52+
migrations.AddIndex(
53+
model_name='discoveredpackagescore',
54+
index=models.Index(fields=['scoring_tool_version'], name='scanpipe_di_scoring_7fa482_idx'),
55+
),
56+
migrations.AddIndex(
57+
model_name='scorecardcheck',
58+
index=models.Index(fields=['check_score'], name='scanpipe_sc_check_s_e189f7_idx'),
59+
),
60+
migrations.AddIndex(
61+
model_name='scorecardcheck',
62+
index=models.Index(fields=['check_name'], name='scanpipe_sc_check_n_1df2b1_idx'),
63+
),
64+
]

scanpipe/models.py

Lines changed: 100 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,8 @@
8888
from rq.exceptions import NoSuchJobError
8989
from rq.job import Job
9090
from rq.job import JobStatus
91+
from scorecode.contrib.django.models import PackageScoreMixin
92+
from scorecode.contrib.django.models import ScorecardChecksMixin
9193
from taggit.managers import TaggableManager
9294
from taggit.models import GenericUUIDTaggedItemBase
9395
from taggit.models import TaggedItemBase
@@ -3932,9 +3934,9 @@ class DiscoveredDependency(
39323934
39333935
3. Dependencies can be either direct or transitive:
39343936
- A **direct dependency** is explicitly declared in a package manifest or
3935-
lockfile.
3937+
lockfile.
39363938
- A **transitive dependency** is not declared directly, but is required by one
3937-
of the project's direct dependencies.
3939+
of the project's direct dependencies.
39383940
39393941
Understanding the distinction between direct and transitive dependencies is
39403942
important for analyzing dependency trees, resolving version conflicts, and
@@ -4760,3 +4762,99 @@ def create_auth_token(sender, instance=None, created=False, **kwargs):
47604762
"""Create an API key token on user creation, using the signal system."""
47614763
if created:
47624764
Token.objects.create(user_id=instance.pk)
4765+
4766+
4767+
class DiscoveredPackageScore(UUIDPKModel, PackageScoreMixin):
4768+
"""Represents a security or quality score for a DiscoveredPackage."""
4769+
4770+
discovered_package = models.ForeignKey(
4771+
DiscoveredPackage,
4772+
related_name="scores",
4773+
help_text=_("The package for which the score is given"),
4774+
on_delete=models.CASCADE,
4775+
editable=False,
4776+
)
4777+
4778+
class Meta:
4779+
verbose_name = "discovered package score"
4780+
verbose_name_plural = "discovered package scores"
4781+
ordering = ["-score"]
4782+
indexes = [
4783+
models.Index(fields=["score"]),
4784+
models.Index(fields=["scoring_tool_version"]),
4785+
]
4786+
4787+
def __str__(self):
4788+
return self.score or str(self.uuid)
4789+
4790+
@classmethod
4791+
def create_from_scorecard_data(
4792+
cls, discovered_package, scorecard_data, scoring_tool="ossf-scorecard"
4793+
):
4794+
"""Create ScoreCard object from scorecard data and discovered package"""
4795+
final_data = {
4796+
"score": scorecard_data.score,
4797+
"scoring_tool_version": scorecard_data.scoring_tool_version,
4798+
"scoring_tool_documentation_url": (
4799+
scorecard_data.scoring_tool_documentation_url
4800+
),
4801+
"score_date": cls.parse_score_date(scorecard_data.score_date),
4802+
}
4803+
4804+
scorecard_object = cls.objects.create(
4805+
**final_data,
4806+
discovered_package=discovered_package,
4807+
scoring_tool=scoring_tool,
4808+
)
4809+
4810+
for check in scorecard_data.checks:
4811+
ScorecardCheck.create_from_data(package_score=scorecard_object, check=check)
4812+
4813+
return scorecard_object
4814+
4815+
@classmethod
4816+
def create_from_package_and_scorecard(cls, scorecard_data, package):
4817+
score_object = cls.create_from_scorecard_data(
4818+
discovered_package=package,
4819+
scorecard_data=scorecard_data,
4820+
scoring_tool="ossf-scorecard",
4821+
)
4822+
return score_object
4823+
4824+
4825+
class ScorecardCheck(UUIDPKModel, ScorecardChecksMixin):
4826+
"""
4827+
Represents an individual check within a Scorecard evaluation for a
4828+
DiscoveredPackageScore.
4829+
"""
4830+
4831+
package_score = models.ForeignKey(
4832+
DiscoveredPackageScore,
4833+
related_name="checks",
4834+
help_text=_("The checks for which the score is given"),
4835+
on_delete=models.CASCADE,
4836+
editable=False,
4837+
)
4838+
4839+
class Meta:
4840+
verbose_name = "scorecard check"
4841+
verbose_name_plural = "scorecard checks"
4842+
ordering = ["-check_score"]
4843+
indexes = [
4844+
models.Index(fields=["check_score"]),
4845+
models.Index(fields=["check_name"]),
4846+
]
4847+
4848+
def __str__(self):
4849+
return self.check_score or str(self.uuid)
4850+
4851+
@classmethod
4852+
def create_from_data(cls, package_score, check):
4853+
"""Create a ScorecardCheck instance from provided data."""
4854+
return cls.objects.create(
4855+
check_name=check.check_name,
4856+
check_score=check.check_score,
4857+
reason=check.reason or "",
4858+
details=check.details or [],
4859+
package_score=package_score,
4860+
)

scanpipe/pipelines/fetch_scores.py

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# SPDX-License-Identifier: Apache-2.0
2+
#
3+
# http://nexb.com and https://github.com/nexB/scancode.io
4+
# The ScanCode.io software is licensed under the Apache License version 2.0.
5+
# Data generated with ScanCode.io is provided as-is without warranties.
6+
# ScanCode is a trademark of nexB Inc.
7+
#
8+
# You may not use this software except in compliance with the License.
9+
# You may obtain a copy of the License at: http://apache.org/licenses/LICENSE-2.0
10+
# Unless required by applicable law or agreed to in writing, software distributed
11+
# under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
12+
# CONDITIONS OF ANY KIND, either express or implied. See the License for the
13+
# specific language governing permissions and limitations under the License.
14+
#
15+
# Data Generated with ScanCode.io is provided on an "AS IS" BASIS, WITHOUT WARRANTIES
16+
# OR CONDITIONS OF ANY KIND, either express or implied. No content created from
17+
# ScanCode.io should be considered or used as legal advice. Consult an Attorney
18+
# for any legal advice.
19+
#
20+
# ScanCode.io is a free software code scanning tool from nexB Inc. and others.
21+
# Visit https://github.com/nexB/scancode.io for support and download.
22+
23+
24+
from scorecode import ossf_scorecard
25+
26+
from scanpipe.models import DiscoveredPackageScore
27+
from scanpipe.pipelines import Pipeline
28+
29+
30+
class FetchScores(Pipeline):
31+
"""
32+
Fetch ScoreCode information for packages.
33+
34+
This pipeline retrieves ScoreCode data for each package in the project
35+
and stores it in the corresponding package instances.
36+
37+
ScoreCode data refers to metadata retrieved from the OpenSSF Scorecard tool,
38+
which evaluates open source packages based on security and quality checks.
39+
This data includes an overall score, individual check results (such as use
40+
of branch protection, fuzzing, dependency updates, etc.), the version of the
41+
scoring tool used, and the date of evaluation
42+
"""
43+
44+
download_inputs = False
45+
is_addon = True
46+
47+
@classmethod
48+
def steps(cls):
49+
return (
50+
cls.check_scorecode_service_availability,
51+
cls.fetch_packages_scorecode_info,
52+
)
53+
54+
def check_scorecode_service_availability(self):
55+
"""Check if the ScoreCode service is configured and available."""
56+
if not ossf_scorecard.is_available():
57+
raise Exception("ScoreCode service is not available.")
58+
59+
def fetch_packages_scorecode_info(self):
60+
"""Fetch ScoreCode information for each of the project's discovered packages."""
61+
for package in self.project.discoveredpackages.all():
62+
if scorecard_data := ossf_scorecard.fetch_scorecard_info(package=package):
63+
DiscoveredPackageScore.create_from_package_and_scorecard(
64+
scorecard_data=scorecard_data,
65+
package=package,
66+
)

0 commit comments

Comments
 (0)