Merge branch 'develop' into #2390

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Author: AyanSinhaMahapatra

CHANGELOG.rst

@@ -24,17 +24,30 @@ Breaking API changes:
    multiple manifests for a single package instance.
 
 
-v21.5.11
+v21.6.2
 --------
 
 Breaking API changes:
 ~~~~~~~~~~~~~~~~~~~~~
 
- - The configure scripts for Linux, macOS and Windows are new. These are now
-   only native scripts (.bat on Windows and .sh on POSIX) and the Python script
-   etc/configure.py has been removed. Use the PYTHON_EXECUTABLE enviroment
-   varibale to point to alternative non-default Python executable. An this on
-   all OSes.
+ - The configure scripts for Linux, macOS and Windows have been entirely
+   refactored and should be considered as new. These are now only native scripts
+   (.bat on Windows and .sh on POSIX) and the Python script etc/configure.py
+   has been removed. Use the PYTHON_EXECUTABLE enviroment variable to point to
+   alternative non-default Python executable and this on all OSes.
+
+Security updates:
+~~~~~~~~~~~~~~~~~
+
+ - Update minimum versions and pinned version of thirdparty dependencies
+   to benefit from latest improvements and sceurity fixes. This includes in
+   particular this issues:
+
+     - pkg:pypi/pygments: (low severity, limited impact) CVE-2021-20270, CVE-2021-27291
+     - pkg:pypi/lxml: (low severity, likely no impact) CVE-2021-28957
+     - pkg:pypi/nltk: (low severity, likely no impact) CVE-2019-14751
+     - pkg:pypi/jinja2: (low severity, likely no impact) CVE-2020-28493, CVE-2019-10906
+     - pkg:pypi/pycryptodome: (high severity) CVE-2018-15560 (dropped since no longer used)
 
 
 Ouputs:

README.rst

@@ -215,13 +215,6 @@ See the NOTICE file and the .ABOUT files that document the origin and license of
 the third-party code used in ScanCode for more details.
 
 
-.. |master-cov| image:: https://codecov.io/gh/nexB/scancode-toolkit/branch/master/graph/badge.svg
-    :target: https://codecov.io/gh/nexB/scancode-toolkit/branch/master
-    :alt: Master branch test coverage (Linux)
-.. |devel-cov| image:: https://codecov.io/gh/nexB/scancode-toolkit/branch/develop/graph/badge.svg
-    :target: https://codecov.io/gh/nexB/scancode-toolkit/branch/develop
-    :alt: Develop branch test coverage (Linux)
-
 .. |master-posix| image:: https://api.travis-ci.org/nexB/scancode-toolkit.png?branch=master
     :target: https://travis-ci.org/nexB/scancode-toolkit
     :alt: Linux Master branch tests status

azure-pipelines.yml

@@ -317,4 +317,33 @@ jobs:
 #          - script: ./etc/release/scancode-test-pip-install-editable.sh
 #            displayName: 'Test pip editable installation'
 
+################################################################################
+# Tests using a plain pip install to get the latest of all wheels
+################################################################################
+
+
+    - template: etc/ci/azure-posix.yml
+      parameters:
+          job_name: ubuntu20_cpython_latest_from_pip
+          image_name: ubuntu-20.04
+          python_versions: ['3.6', '3.7', '3.8', '3.9']
+          test_suites:
+              all: bin/pip install --force-reinstall --upgrade -e .[dev] && bin/pytest -n 2 -vvs tests/scancode/test_cli.py
+
+    - template: etc/ci/azure-win.yml
+      parameters:
+          job_name: win2019_cpython_latest_from_pip
+          image_name: windows-2019
+          python_versions: ['3.6', '3.7', '3.8', '3.9']
+          test_suites:
+              all: Scripts\pip install --force-reinstall --upgrade -e .[dev] &&  Scripts\pytest -n 2 -vvs  tests\scancode\test_cli.py
+
+    - template: etc/ci/azure-posix.yml
+      parameters:
+          job_name: macos1015_cpython_latest_from_pip
+          image_name: macos-10.15
+          python_versions: ['3.6', '3.7', '3.8', '3.9']
+          test_suites:
+              all: bin/pip install --force-reinstall --upgrade -e .[dev] && bin/pytest -n 2 -vvs  tests/scancode/test_cli.py
+
 

docs/source/getting-started/install.rst

@@ -120,12 +120,12 @@ Installation on Linux and Mac
 Get the Scancode Toolkit tarball archive of a specific version and your
 operating system by going to the `project releases page <https://github.com/nexB/scancode-toolkit/releases/>`_
 
-For example, Version 21.3.31 archive can be obtained from
-`Toolkit release 21.3.31 <https://github.com/nexB/scancode-toolkit/releases/tag/v21.3.31>`_
+For example, Version 21.6.2 archive can be obtained from
+`Toolkit release 21.6.2 <https://github.com/nexB/scancode-toolkit/releases/tag/v21.6.1>`_
 under assets options. Download the archive for your operating systen and extract
 the archive from command line::
 
-    tar -xvf scancode-toolkit-21.3.31_py36-linux.tar.xz
+    tar -xvf scancode-toolkit-21.6.2_py36-linux.tar.xz
 
 
 Or, on Linux, right click and select "Extract Here".
@@ -292,7 +292,7 @@ Here, ``main`` branch has the latest release of Scancode-Toolkit. You can also c
 of the following:
 
 - Branches (Locally created or already present) [Example - ``main``]
-- Tags (essentially version numbers) [Example - ``v21.2.25``, ``v21.3.31``]
+- Tags (essentially version numbers) [Example - ``v21.2.25``, ``v21.6.1``]
 - Commits (use the shortened commit hash) [Example - ``4502055``, ``f276398``]
 
 

etc/release/bootstrap.py

@@ -52,6 +52,10 @@
     show_default=True,
     help='OS(ses) to use for this build: one of linux, mac or windows.',
 )
+@click.option('-l', '--latest-version',
+    is_flag=True,
+    help='Get the latest version of all packages, ignoring version specifiers.',
+)
 @click.option('--sync-dejacode',
     is_flag=True,
     help='Synchronize packages with DejaCode.',
@@ -67,19 +71,23 @@ def bootstrap(
     python_version,
     operating_system,
     with_deps,
+    latest_version,
     sync_dejacode,
     build_remotely=False,
 ):
     """
     Boostrap a thirdparty Python packages directory from pip requirements.
 
     Fetch or build to THIRDPARTY_DIR all the wheels and source distributions for
-    the pip `--requirement-file` requirements FILE(s). Build wheels compatible
-    with all the provided `--python-version` PYVER(s) and `--operating_system`
+    the pip ``--requirement-file`` requirements FILE(s). Build wheels compatible
+    with all the provided ``--python-version`` PYVER(s) and ```--operating_system``
     OS(s) defaulting to all supported combinations. Create or fetch .ABOUT and
     .LICENSE files.
 
-    Sources and wheels are first fetched from PyPI, then our remote repository.
+    Optionally ignore version specifiers and use the ``--latest-version`` 
+    of everything.
+
+    Sources and wheels are fetched with attempts first from PyPI, then our remote repository.
     If missing wheels are built as needed.
     """
     # rename variables for clarity since these are lists
@@ -99,8 +107,9 @@ def bootstrap(
     for req_file in requirements_files:
         nvs = utils_thirdparty.load_requirements(
             requirements_file=req_file, force_pinned=False)
-
         required_name_versions.update(nvs)
+    if latest_version:
+        required_name_versions = set((name, None) for name, _ver in required_name_versions)
 
     print(f'PROCESSING {len(required_name_versions)} REQUIREMENTS in {len(requirements_files)} FILES')
 

etc/release/utils_thirdparty.py

@@ -2685,12 +2685,20 @@ def get_romp_pyos_options(
     python_versions=PYTHON_VERSIONS,
     operating_systems=PLATFORMS_BY_OS,
 ):
-    python_dot_versions = ['.'.join(pv) for pv in python_versions]
-    pyos_options = sorted(set(itertools.chain.from_iterable(
-        ('--version', ver) for ver in python_dot_versions)))
+    """
+    Return a list of CLI options for romp
+    For example:
+    >>> expected = ['--version', '3.6', '--version', '3.7', '--version', '3.8',
+    ... '--version', '3.9', '--platform', 'linux', '--platform', 'macos',
+    ... '--platform', 'windows']
+    >>> assert get_romp_pyos_options() == expected
+    """
+    python_dot_versions = ['.'.join(pv) for pv in sorted(set(python_versions))]
+    pyos_options = list(itertools.chain.from_iterable(
+        ('--version', ver) for ver in python_dot_versions))
 
-    pyos_options += sorted(set(itertools.chain.from_iterable(
-        ('--platform' , plat) for plat in operating_systems)))
+    pyos_options += list(itertools.chain.from_iterable(
+        ('--platform' , plat) for plat in sorted(set(operating_systems))))
 
     return pyos_options
 

etc/thirdparty/bsd-simplified.LICENSE

@@ -1,7 +1,20 @@
-Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
 
-Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+Redistributions of source code must retain the above copyright notice, this list
+of conditions and the following disclaimer.
 
-Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+Redistributions in binary form must reproduce the above copyright notice, this
+list of conditions and the following disclaimer in the documentation and/or
+other materials provided with the distribution.
 
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

etc/thirdparty/virtualenv.pyz

@@ -1,7 +1,7 @@
 about_resource: virtualenv.pyz
 name: get-virtualenv
-version: 20.4.2
-download_url: https://github.com/pypa/get-virtualenv/raw/20.4.2/public/virtualenv.pyz
+version: 20.4.7
+download_url: https://github.com/pypa/get-virtualenv/raw/20.4.7/public/virtualenv.pyz
 description: virtualenv is a tool to create isolated Python environments.
 homepage_url: https://github.com/pypa/virtualenv
 license_expression: lgpl-2.1-plus AND (bsd-new OR apache-2.0) AND mit AND python AND bsd-new
@@ -10,26 +10,4 @@ copyright: Copyright (c) The Python Software Foundation and others
 redistribute: yes
 attribute: yes
 track_changes: yes
-package_url: pkg:github/pypa/[email protected]#public/virtualenv.pyz
-licenses:
-  - key: apache-2.0
-    name: Apache License 2.0
-    file: apache-2.0.LICENSE
-  - key: mit
-    name: MIT License
-    file: mit.LICENSE
-  - key: bsd-simplified
-    name: BSD-2-Clause
-    file: bsd-simplified.LICENSE
-  - key: bsd-new
-    name: BSD-3-Clause
-    file: bsd-new.LICENSE
-  - key: python
-    name: Python Software Foundation License v2
-    file: python.LICENSE
-  - key: lgpl-2.1-plus
-    name: GNU Lesser General Public License 2.1 or later
-    file: lgpl-2.1-plus.LICENSE
-  - key: isc
-    name: ISC License
-    file: isc.LICENSE
+package_url: pkg:github/pypa/[email protected]#public/virtualenv.pyz

etc/thirdparty/virtualenv.pyz.ABOUT

@@ -1,12 +1,11 @@
 aboutcode-toolkit==6.0.0
 apipkg==1.5
-codecov==2.1.11
-coverage==5.3.1
-execnet==1.7.1
+coverage==5.5
+execnet==1.8.1
 iniconfig==1.1.1
 py==1.10.0
-pytest==6.2.1
+pytest==6.2.4
 pytest-cov==2.11.1
 pytest-forked==1.3.0
-pytest-rerunfailures==9.1.1
-pytest-xdist==2.2.0
+pytest-rerunfailures==10.0
+pytest-xdist==2.2.1