Skip to content

Commit a87a4d5

Browse files
AyanSinhaMahapatraAyanSinhaMahapatra
committed
Merge branch 'develop' into license-detection-improvements-and-review
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
2 parents 1bad407 + 94d4fe6 commit a87a4d5

File tree

42 files changed

+17450
-2087
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+17450
-2087
lines changed

‎.github/workflows/about-files-ci.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,10 +16,10 @@ jobs:
1616

1717
steps:
1818
- name: Checkout code
19-
uses: actions/checkout@v2
19+
uses: actions/checkout@v3
2020

2121
- name: Set up Python ${{ matrix.python-version }}
22-
uses: actions/setup-python@v2
22+
uses: actions/setup-python@v4
2323
with:
2424
python-version: ${{ matrix.python-version }}
2525

‎.github/workflows/docs-ci.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,10 +16,10 @@ jobs:
1616

1717
steps:
1818
- name: Checkout code
19-
uses: actions/checkout@v2
19+
uses: actions/checkout@v3
2020

2121
- name: Set up Python ${{ matrix.python-version }}
22-
uses: actions/setup-python@v2
22+
uses: actions/setup-python@v4
2323
with:
2424
python-version: ${{ matrix.python-version }}
2525

‎.github/workflows/scancode-release.yml

Lines changed: 23 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -37,10 +37,10 @@ jobs:
3737
pyver: ["3.7", "3.8", "3.9", "3.10", "3.11"]
3838

3939
steps:
40-
- uses: actions/checkout@v2
40+
- uses: actions/checkout@v3
4141

4242
- name: Set up Python on ${{ matrix.pyver }}
43-
uses: actions/setup-python@v1
43+
uses: actions/setup-python@v4
4444
with:
4545
python-version: ${{ matrix.pyver }}
4646

@@ -69,10 +69,10 @@ jobs:
6969
fail-fast: true
7070

7171
steps:
72-
- uses: actions/checkout@v2
72+
- uses: actions/checkout@v3
7373

7474
- name: Set up Python
75-
uses: actions/setup-python@v1
75+
uses: actions/setup-python@v4
7676
with:
7777
python-version: "3.10"
7878

@@ -103,10 +103,10 @@ jobs:
103103
pyver: ["3.7", "3.8", "3.9", "3.10", "3.11"]
104104

105105
steps:
106-
- uses: actions/checkout@v2
106+
- uses: actions/checkout@v3
107107

108108
- name: Set up Python on ${{ matrix.pyver }}
109-
uses: actions/setup-python@v1
109+
uses: actions/setup-python@v4
110110
with:
111111
python-version: ${{ matrix.pyver }}
112112

@@ -137,10 +137,10 @@ jobs:
137137
pyver: ["3.7", "3.8", "3.9", "3.10", "3.11"]
138138

139139
steps:
140-
- uses: actions/checkout@v2
140+
- uses: actions/checkout@v3
141141

142142
- name: Set up Python on ${{ matrix.pyver }}
143-
uses: actions/setup-python@v1
143+
uses: actions/setup-python@v4
144144
with:
145145
python-version: ${{ matrix.pyver }}
146146

@@ -171,10 +171,10 @@ jobs:
171171
pyver: ["3.7", "3.8", "3.9", "3.10", "3.11"]
172172

173173
steps:
174-
- uses: actions/checkout@v2
174+
- uses: actions/checkout@v3
175175

176176
- name: Set up Python on ${{ matrix.pyver }}
177-
uses: actions/setup-python@v1
177+
uses: actions/setup-python@v4
178178
with:
179179
python-version: ${{ matrix.pyver }}
180180

@@ -202,10 +202,10 @@ jobs:
202202
fail-fast: true
203203

204204
steps:
205-
- uses: actions/checkout@v2
205+
- uses: actions/checkout@v3
206206

207207
- name: Set up Python
208-
uses: actions/setup-python@v1
208+
uses: actions/setup-python@v4
209209
with:
210210
python-version: "3.10"
211211

@@ -239,10 +239,10 @@ jobs:
239239
pyver: ["3.7", "3.8", "3.9", "3.10", "3.11"]
240240

241241
steps:
242-
- uses: actions/checkout@v2
242+
- uses: actions/checkout@v3
243243

244244
- name: Set up Python ${{ matrix.pyver }} on ${{ matrix.os }}
245-
uses: actions/setup-python@v1
245+
uses: actions/setup-python@v4
246246
with:
247247
python-version: ${{ matrix.pyver }}
248248

@@ -285,10 +285,10 @@ jobs:
285285
pyver: ["3.7", "3.8", "3.9", "3.10", "3.11"]
286286

287287
steps:
288-
- uses: actions/checkout@v2
288+
- uses: actions/checkout@v3
289289

290290
- name: Set up Python ${{ matrix.pyver }} on ${{ matrix.os }}
291-
uses: actions/setup-python@v1
291+
uses: actions/setup-python@v4
292292
with:
293293
python-version: ${{ matrix.pyver }}
294294

@@ -330,10 +330,10 @@ jobs:
330330
pyver: ["3.7", "3.8", "3.9", "3.10", "3.11"]
331331

332332
steps:
333-
- uses: actions/checkout@v2
333+
- uses: actions/checkout@v3
334334

335335
- name: Set up Python ${{ matrix.pyver }} on ${{ matrix.os }}
336-
uses: actions/setup-python@v1
336+
uses: actions/setup-python@v4
337337
with:
338338
python-version: ${{ matrix.pyver }}
339339

@@ -371,10 +371,10 @@ jobs:
371371
pyver: ["3.7", "3.8", "3.9", "3.10", "3.11"]
372372

373373
steps:
374-
- uses: actions/checkout@v2
374+
- uses: actions/checkout@v3
375375

376376
- name: Set up Python ${{ matrix.pyver }} on ${{ matrix.os }}
377-
uses: actions/setup-python@v1
377+
uses: actions/setup-python@v4
378378
with:
379379
python-version: ${{ matrix.pyver }}
380380

@@ -412,10 +412,10 @@ jobs:
412412
pyver: ["3.7", "3.8", "3.9", "3.10", "3.11"]
413413

414414
steps:
415-
- uses: actions/checkout@v2
415+
- uses: actions/checkout@v3
416416

417417
- name: Set up Python ${{ matrix.pyver }} on ${{ matrix.os }}
418-
uses: actions/setup-python@v1
418+
uses: actions/setup-python@v4
419419
with:
420420
python-version: ${{ matrix.pyver }}
421421

@@ -577,7 +577,7 @@ jobs:
577577

578578
steps:
579579
- name: Set up Python
580-
uses: actions/setup-python@v1
580+
uses: actions/setup-python@v4
581581
with:
582582
python-version: 3.8
583583

‎.readthedocs.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,11 @@
55
# Required
66
version: 2
77

8+
# Build PDF & ePub
9+
formats:
10+
- epub
11+
- pdf
12+
813
# Where the Sphinx conf.py file is located
914
sphinx:
1015
configuration: docs/source/conf.py

‎CHANGELOG.rst

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,28 @@ v32.1.0 (next, roadmap)
4242
See https://github.com/nexB/scancode-toolkit/issues/1745
4343

4444

45+
v32.0.4 - 2023-06-07
46+
---------------------
47+
48+
This is a minor bugfix release with the following updates:
49+
50+
- Fixes a performance issue issue arising out of license detection
51+
on files happening in a single-threaded process_codebase step when the
52+
license CLI option is disabled for a package scan.
53+
Reference: https://github.com/nexB/scancode-toolkit/pull/3423
54+
55+
v32.0.3 - 2023-06-06
56+
---------------------
57+
58+
This is a minor bugfix release with the following updates:
59+
60+
- We were missing scancode-toolkit-mini releases from v32.0.0rc2 and
61+
also the scancode-toolkit release wheels including and after v32.0.0rc2 were
62+
actually scancode-toolkit-mini releases.
63+
Reference: https://github.com/nexB/scancode-toolkit/issues/3421
64+
65+
- Updated github actions, for more details see https://github.com/nexB/skeleton/issues/75
66+
4567
v32.0.2 - 2023-05-26
4668
---------------------
4769

‎docs/source/conf.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,3 +104,9 @@
104104
.. role:: img-title-para
105105
106106
"""
107+
108+
# -- Options for LaTeX output -------------------------------------------------
109+
110+
latex_elements = {
111+
'classoptions': ',openany,oneside'
112+
}

‎setup-mini.cfg

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[metadata]
2-
name = scancode-toolkit
3-
version = 32.0.2
2+
name = scancode-toolkit-mini
3+
version = 32.0.4
44
license = Apache-2.0 AND CC-BY-4.0 AND LicenseRef-scancode-other-permissive AND LicenseRef-scancode-other-copyleft
55

66
# description must be on ONE line https://github.com/pypa/setuptools/issues/1390

‎setup.cfg

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[metadata]
22
name = scancode-toolkit
3-
version = 32.0.2
3+
version = 32.0.4
44
license = Apache-2.0 AND CC-BY-4.0 AND LicenseRef-scancode-other-permissive AND LicenseRef-scancode-other-copyleft
55

66
# description must be on ONE line https://github.com/pypa/setuptools/issues/1390

‎src/packagedcode/build.py

Lines changed: 14 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,8 @@
1414

1515
from commoncode import fileutils
1616

17+
from licensedcode.cache import build_spdx_license_expression
18+
from licensedcode.cache import get_cache
1719
from licensedcode.tokenize import query_tokenizer
1820
from licensedcode.detection import detect_licenses
1921
from licensedcode.detection import get_unknown_license_detection
@@ -115,13 +117,17 @@ def assemble(cls, package_data, resource, codebase, package_adder):
115117
if TRACE:
116118
logger_debug(f"BaseStarlarkManifestHandler.assemble: package_data: {package_data.to_dict()}")
117119

118-
detections, expression = get_license_detections_and_expression(
119-
package=package_data,
120-
resource=resource,
121-
codebase=codebase,
122-
)
123-
package.license_detections = detections
124-
package.declared_license_expression = expression
120+
package.license_detections, package.declared_license_expression = \
121+
get_license_detections_and_expression(
122+
package=package_data,
123+
resource=resource,
124+
codebase=codebase,
125+
)
126+
if package.declared_license_expression:
127+
package.declared_license_expression_spdx = str(build_spdx_license_expression(
128+
license_expression=package.declared_license_expression,
129+
licensing=get_cache().licensing,
130+
))
125131

126132
cls.assign_package_to_resources(
127133
package=package,
@@ -132,6 +138,7 @@ def assemble(cls, package_data, resource, codebase, package_adder):
132138

133139
yield package
134140

141+
135142
# we yield this as we do not want this further processed
136143
yield resource
137144

‎src/packagedcode/licensing.py

Lines changed: 6 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ def logger_debug(*args):
5656
return logger.debug(' '.join(isinstance(a, str) and a or repr(a) for a in args))
5757

5858

59-
def add_referenced_license_matches_for_package(resource, codebase, no_licenses):
59+
def add_referenced_license_matches_for_package(resource, codebase):
6060
"""
6161
Return an updated ``resource`` saving it in place, after adding new license
6262
detections to the package manifests detected in this resource, following their
@@ -106,13 +106,7 @@ def add_referenced_license_matches_for_package(resource, codebase, no_licenses):
106106
if not referenced_resource:
107107
continue
108108

109-
if no_licenses:
110-
referenced_license_detections = get_license_detection_mappings(
111-
location=referenced_resource.location
112-
)
113-
114-
else:
115-
referenced_license_detections = referenced_resource.license_detections
109+
referenced_license_detections = referenced_resource.license_detections
116110

117111
if referenced_license_detections:
118112
modified = True
@@ -160,7 +154,7 @@ def add_referenced_license_matches_for_package(resource, codebase, no_licenses):
160154
yield resource
161155

162156

163-
def add_referenced_license_detection_from_package(resource, codebase, no_licenses):
157+
def add_referenced_license_detection_from_package(resource, codebase):
164158
"""
165159
Return an updated ``resource`` saving it in place, after adding new license
166160
matches (licenses and license_expressions) following their Rule
@@ -209,7 +203,6 @@ def add_referenced_license_detection_from_package(resource, codebase, no_license
209203
sibling_license_detections, _le = get_license_detections_from_sibling_file(
210204
resource=root_resource,
211205
codebase=codebase,
212-
no_licenses=no_licenses,
213206
)
214207
if TRACE:
215208
logger_debug(
@@ -278,12 +271,10 @@ def add_referenced_license_detection_from_package(resource, codebase, no_license
278271
yield resource
279272

280273

281-
def add_license_from_sibling_file(resource, codebase, no_licenses):
274+
def add_license_from_sibling_file(resource, codebase):
282275
"""
283276
Given a resource and it's codebase object, assign licenses to the package
284277
detections in that resource, from the sibling files of it.
285-
286-
If `no_license` is True, then license scan (for resources) is disabled.
287278
"""
288279
if TRACE:
289280
logger_debug(f'packagedcode.licensing: add_license_from_sibling_file: resource: {resource.path}')
@@ -303,7 +294,6 @@ def add_license_from_sibling_file(resource, codebase, no_licenses):
303294
license_detections, license_expression = get_license_detections_from_sibling_file(
304295
resource=resource,
305296
codebase=codebase,
306-
no_licenses=no_licenses,
307297
)
308298
if not license_detections:
309299
return
@@ -333,13 +323,11 @@ def is_legal_or_readme(resource):
333323
return False
334324

335325

336-
def get_license_detections_from_sibling_file(resource, codebase, no_licenses):
326+
def get_license_detections_from_sibling_file(resource, codebase):
337327
"""
338328
Return `license_detections`, a list of LicenseDetection objects and a
339329
`license_expression`, given a resource and it's codebase object, from
340330
the sibling files of the resource.
341-
342-
If `no_license` is True, then license scan (for resources) is disabled.
343331
"""
344332
siblings = []
345333

@@ -357,15 +345,7 @@ def get_license_detections_from_sibling_file(resource, codebase, no_licenses):
357345

358346
license_detections = []
359347
for sibling in siblings:
360-
if no_licenses:
361-
detections = get_license_detection_mappings(
362-
location=sibling.location,
363-
analysis=DetectionCategory.PACKAGE_ADD_FROM_SIBLING_FILE.value,
364-
post_scan=True,
365-
)
366-
license_detections.extend(detections)
367-
else:
368-
license_detections.extend(sibling.license_detections)
348+
license_detections.extend(sibling.license_detections)
369349

370350
if not license_detections:
371351
return [], None

0 commit comments

Comments
 (0)