Merge branch 'develop' into doc-update-licenses

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Author: AyanSinhaMahapatra

AUTHORS.rst

@@ -44,6 +44,7 @@ The following organizations or individuals have contributed to ScanCode:
 - Lemo Shi @lemoshi
 - Li Ha @linexb
 - Mankaran Singh @MankaranSingh
+- Marc-Etienne Vargenau @vargenau
 - Martin Petkov @MartinPetkov
 - Maximilian Huber @maxhbr 
 - Michael Herzog @mjherzog
@@ -78,6 +79,7 @@ The following organizations or individuals have contributed to ScanCode:
 - Sebastian Schuberth @sschuberth
 - Shankhadeep Dey @Iamshankhadeep
 - Sharikzama @ZamaSharik
+- Shrijal Acharya @OctoPie23
 - Shivam Chauhan @chashiv
 - Shivam Sandbhor @sbs2001
 - Steven Esser @majurg

CHANGELOG.rst

@@ -63,11 +63,17 @@ Package detection:
   manifest data parsing code outside of the scancode-toolkit context in other
   libraries.
 
+- The PackageData model now includes a ``holder`` field, which is populated with
+  holder data extracted from the copyright field if copyright data is present,
+  otherwise it remains empty.
+
+  https://github.com/nexB/scancode-toolkit/issues/3290
+
 
 License detection:
 ~~~~~~~~~~~~~~~~~~~
 
-- The SPDX license list has been updated to the latest v3.19
+- The SPDX license list has been updated to the latest v3.20
 
 - This is a major update to license detection where we now combine one or more
   license matches in a larger license detection. This approach improves the
@@ -78,11 +84,9 @@ License detection:
 - There is a new ``license_detections`` codebase level attribute with all the
   unique license detections in the whole scan, both in resources and packages.
   This has the 3 attributes also present in package/resource level license
-  detections: ``license_expression``, ``matches`` and ``detection_log`` and has
-  two additional attributes:
-
-  - ``identifier``: which is the ``license_expression`` with an UUID created out
-    of the detection contents and is the same for same detections.
+  detections: ``license_expression``, ``identifier`` and ``detection_log``
+  (present optionally if the ``--license-diagnostics`` option is enabled) with
+  an additional attribute:
 
   - ``count``: Number of times in the codebase this unique license detection
     was encountered.
@@ -91,15 +95,14 @@ License detection:
 
   - The ``licenses`` attribute is deleted.
 
-  - A new ``for_license_detections`` attribute is aded which references the codebase
-    level unique license detections, and this is a list of ``identifer`` strings from
-    the codebase level license detections it references.
-  
   - A new ``license_detections`` attribute contains license detections in that file.
-    This object has three attributes: ``license_expression``, ``detection_log``
+    This object has three attributes: ``license_expression``, ``identifier``
     and ``matches``. ``matches`` is a list of license matches and is roughly
     the same as  ``licenses`` in the previous version with additional structure
-    changes detailed below.
+    changes detailed below. Identifier is the detected license-expression with an
+    UUID generated from the content of ``matches`` such that this is unique for
+    unique detections. We also have another attribute ``detection_log`` with
+    diagnostics information if the ``--license-diagnostics`` option is enabled.
 
   - A new attribute ``license_clues`` contains license matches with the
     same data structure as the ``matches`` attribute in ``license_detections``.
@@ -140,13 +143,14 @@ License detection:
   avoiding nesting. See `license updates doc <https://scancode-toolkit.readthedocs.io/en/latest/explanations/license-detection-reference.html#licensematch-result-data>`_
   for examples and details.
 
-- There are new and codebase level attributes default with `--licenses` to report
+- There are new and codebase level attributes with ``--license-references`` to report
   reference license metadata and texts once for each license matched across the
   scan; we now have two codebase level attributes: ``license_references`` and
   ``license_rule_references`` that list unique detected license and license rules.
   for examples and details. This reference data is also removed from license matches
   in all levels i.e. from codebase, package and resource level license detections and
-  resource level license clues.
+  resource level license clues, irrespective of this CLI option being used, i.e. default
+  with ``--licenses``.
   See `license updates documentation <https://scancode-toolkit.readthedocs.io/en/latest/explanations/license-detection-reference.html#comparision-before-after-license-references>`_
 
 - We replaced the ``scancode --reindex-licenses`` command line option with a
@@ -166,7 +170,7 @@ License detection:
 
   - See https://github.com/nexB/scancode-toolkit/issues/480 for more details.
 
-- We combined the licensedata file and text file of each license in a single
+- We combined the license data file and text file of each license in a single
   file with a .LICENSE extension. The .yml data file is now included at the
   top of each .LICENSE file as "YAML frontmatter". The same applies to license
   rules and their .RULE and .yml files. This halves the number of data files

CONTRIBUTING.rst

@@ -3,7 +3,7 @@ Contributing
 ============
 
 Contributions are welcome and appreciated!
-Every little bit helps, and credit will always be given.
+Every little bit helps, and a credit will always be given.
 
 .. _issues : https://github.com/nexB/scancode-toolkit/issues
 __ issues_
@@ -14,15 +14,15 @@ check `easy issues <https://github.com/nexB/scancode-toolkit/labels/easy>`_
 When contributing to ScanCode (such as code, bugs, documentation, etc.) you
 agree to the Developer `Certificate of Origin <http://developercertificate.org/>`_
 and the ScanCode license (see the `NOTICE <https://github.com/nexB/scancode-toolkit/blob/develop/NOTICE>`_ file).
-The same approach is used by the Linux Kernel developers and several other projects.
+The same approach is used by Linux Kernel developers and several other projects.
 
 For commits, it is best to simply add a line like this to your commit message,
 with your name and email::
 
     Signed-off-by: Jane Doe <[email protected]>
 
 Please try to write a good commit message, see `good commit message wiki
-<https://aboutcode.readthedocs.io/en/latest/contributing/writing_good_commit_messages.html>` for
+<https://aboutcode.readthedocs.io/en/latest/contributing/writing_good_commit_messages.html>`_ for
 details. In particular use the imperative for your commit subject: think that
 you are giving an order to the codebase to update itself.
 
@@ -35,7 +35,7 @@ To send feedback or ask a question, `file an issue <issues_>`_
 If you are proposing a feature:
 
 * Explain how it would work.
-* Keep the scope simple possible to make it easier to implement.
+* Keep the scope as simple as possible to make it easier to implement.
 * Remember that your contributions are welcomed to implement this feature!
 
 
@@ -54,23 +54,21 @@ For other questions, discussions, and chats, we have:
   This is a busy place with a lot of CI and commit notifications that makes
   actual chat sometimes difficult!
 
-- a mailing list at `sourceforge <https://lists.sourceforge.net/lists/listinfo/aboutcode-discuss>`_
-
 - a Gitter channel to discuss Documentation at https://gitter.im/aboutcode-org/gsod-season-of-docs
 
 Bug reports
 ===========
 
 When `reporting a bug`__ please include:
 
-* Your operating system name, version and architecture (32 or 64 bits).
+* Your operating system name, version, and architecture (32 or 64 bits).
 * Your Python version.
 * Your ScanCode version.
 * Any additional details about your local setup that might be helpful to
   diagnose this bug.
 * Detailed steps to reproduce the bug, such as the commands you ran and a link
   to the code you are scanning.
-* The errors messages or failure trace if any.
+* The error messages or failure trace if any.
 * If helpful, you can add a screenshot as an issue attachment when relevant or
   some extra file as a link to a `Gist <https://gist.github.com>`_.
 
@@ -83,11 +81,11 @@ any other general upgrades, etc. Even a minor typo fix is welcomed.
 
 If something is missing in the documentation or if you found some part confusing,
 please file an issue with your suggestions for improvement. Use the “Documentation Improvement”
-template. Your help and contributions makes ScanCode docs better, we love hearing from you!
+template. Your help and contribution make ScanCode docs better, we love hearing from you!
 
 The ScanCode documentation is hosted at `scancode-toolkit.readthedocs.io <https://scancode-toolkit.readthedocs.io/en/latest/>`_.
 
-If you want to contribute to Scancode Dcoumentation, you'll find `this guide here <https://scancode-toolkit.readthedocs.io/en/latest/contribute/contrib_doc.html>`_ helpful.
+If you want to contribute to Scancode Documentation, you'll find `this guide here <https://scancode-toolkit.readthedocs.io/en/latest/contribute/contrib_doc.html>`_ helpful.
 
 Development
 ===========
@@ -124,11 +122,13 @@ To set up ScanCode for local development:
 3. Create a branch for local development::
 
     git checkout -b name-of-your-bugfix-or-feature
+    
+4. Check out the Contributing to Code Development `documentation <https://scancode-toolkit.readthedocs.io/en/stable/contribute/contrib_dev.html>`_, as it contains more in-depth guide for contributing code and documentation.
 
-4. To configure your local environment for development, locate to the main
-   directory of the local repository, run the configure script.
+5. To configure your local environment for development, locate to the main
+   directory of the local repository, and run the configure script.
    The configure script creates an isolated Python `virtual environment` in
-   your checkout directory, the Python `pip` tool, and installs the thirdparty
+   your checkout directory, the Python `pip` tool, and installs the third-party
    libraries (from the `thirdparty/ directory`), setup the paths, etc.
    See https://virtualenv.pypa.io/en/latest/ for more details. 
 
@@ -157,10 +157,20 @@ To set up ScanCode for local development:
    When you pull new code from git, rerun ./configure
 
 
-5. Now you can make your code changes in your local clone.
+6. Now you can make your code changes in your local clone.
    Please create new unit tests for your code. We love tests!
 
-6. When you are done with your changes, run all the tests.
+7. An update to the ``CHANGELOG`` is required if any important changes are made that needs to be communicated such as:
+
+   * Changes in the API.
+
+   * Addition or deletion of CLI options.
+
+   * Addition of any new feature or any other miscellaneous changes to the program.
+   
+8. If there is a code change, a significant document, or any other changes, you must update the ``AUTHORS`` to include your own name.
+
+9. When you are done with your changes, run all the tests.
    Use this command::
 
         py.test
@@ -169,7 +179,7 @@ To set up ScanCode for local development:
 
        py.test -n6
 
-   If you are running this on a RedHat based OS you may come across this
+   If you are running this on a RedHat-based OS you may come across this
    failure::
 
        OSError: libbz2.so.1.0: cannot open shared object file: No such file or directory
@@ -182,18 +192,18 @@ To set up ScanCode for local development:
 
    See `this issue <https://github.com/nexB/scancode-toolkit/issues/443>`_ for more information.
 
-7. Check the status of your local repository before commit, regarding files changed::
+10. Check the status of your local repository before committing, regarding files changed::
 
-    git status
+     git status
 
 
-8. Commit your changes and push your branch to your GitHub fork::
+11. Commit your changes and push your branch to your GitHub fork::
 
-    git add <file-changed-1> <file-changed-2> <file-changed-3>
-    git commit -m "Your detailed description of your changes." --signoff
-    git push <repository-alias-name> name-of-your-bugfix-or-feature
+     git add <file-changed-1> <file-changed-2> <file-changed-3>
+     git commit -m "Your detailed description of your changes." --signoff
+     git push <repository-alias-name> name-of-your-bugfix-or-feature
 
-9. Submit a pull request through the GitHub website for this branch.
+12. Submit a pull request through the GitHub website for this branch.
 
 
 Pull Request Guidelines
@@ -205,7 +215,7 @@ create a pull request. You can add new commits to your branch as needed.
 For merging, your request would need to:
 
 1. Include unit tests that are passing (run ``py.test``).
-2. Update documentation as needed for new API, functionality etc.
+2. Update documentation as needed for new API, functionality, etc.
 3. Add a note to ``CHANGELOG.rst`` about the changes.
 4. Add your name to ``AUTHORS.rst``.
 
@@ -228,3 +238,4 @@ To run tests in parallel on eight processors::
 To run tests verbosely, displaying all print statements to terminal::
 
     py.test  -vvs
+

azure-pipelines.yml

@@ -91,15 +91,6 @@ jobs:
                 venv/bin/pytest -vvs --test-suite=plugins \
                   tests/licensedcode/test_additional_license.py
 
-    - template: etc/ci/azure-posix.yml
-      parameters:
-          job_name: ubuntu18_cpython
-          image_name: ubuntu-18.04
-          python_versions: ['3.8', '3.9', '3.10', '3.11']
-          python_architecture: x64
-          test_suites:
-              all: venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py --reruns 2
-
     - template: etc/ci/azure-posix.yml
       parameters:
           job_name: ubuntu20_cpython
@@ -147,9 +138,9 @@ jobs:
 
     - template: etc/ci/azure-win.yml
       parameters:
-          job_name: win2019_cpython_2
-          image_name: windows-2019
-          python_versions: ['3.9', '3.10', '3.11']
+          job_name: win2022_cpython
+          image_name: windows-2022
+          python_versions: ['3.8', '3.9', '3.10', '3.11']
           python_architecture: x64
           test_suites:
               all: venv\Scripts\pytest -n 2 -vvs tests\scancode\test_cli.py --reruns 2
@@ -206,14 +197,6 @@ jobs:
           test_suites:
               all: venv/bin/pip install --upgrade-strategy eager --force-reinstall --upgrade -e .[testing] && venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py
 
-    - template: etc/ci/azure-posix.yml
-      parameters:
-          job_name: ubuntu18_cpython_latest_from_pip
-          image_name: ubuntu-18.04
-          python_versions: ['3.8', '3.9', '3.10', '3.11']
-          test_suites:
-              all: venv/bin/pip install --upgrade-strategy eager --force-reinstall --upgrade -e .[testing] && venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py
-
     - template: etc/ci/azure-posix.yml
       parameters:
           job_name: ubuntu20_cpython_latest_from_pip

configure

@@ -159,7 +159,7 @@ PROD_REQUIREMENTS="scancode_toolkit*.whl"
 VIRTUALENV_DIR=venv
 
 # Cleanable files and directories to delete with the --clean option
-CLEANABLE="build venv"
+CLEANABLE="build dist venv .cache .eggs"
 
 # extra  arguments passed to pip
 PIP_EXTRA_ARGS=" "
@@ -175,6 +175,13 @@ CFG_ROOT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
 CFG_BIN_DIR=$CFG_ROOT_DIR/$VIRTUALENV_DIR/bin
 
 
+################################
+# Install with or without and index. With "--no-index" this is using only local wheels
+# This is an offline mode with no index and no network operations
+# NO_INDEX="--no-index "
+NO_INDEX=""
+
+
 ################################
 # Thirdparty package locations and index handling
 # Find packages from the local thirdparty directory if present
@@ -308,6 +315,7 @@ while getopts :-: optchar; do
     esac
 done
 
+
 PIP_EXTRA_ARGS="$PIP_EXTRA_ARGS"
 
 find_python

configure.bat

@@ -35,7 +35,7 @@ set "REL_REQUIREMENTS=--requirement etc/scripts/requirements.txt"
 set "VIRTUALENV_DIR=venv"
 
 @rem # Cleanable files and directories to delete with the --clean option
-set "CLEANABLE=build venv"
+set "CLEANABLE=build dist venv .cache .eggs"
 
 @rem # extra  arguments passed to pip
 set "PIP_EXTRA_ARGS= "

docs/source/conf.py

@@ -28,16 +28,16 @@
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
-'sphinx.ext.intersphinx',
+    "sphinx.ext.intersphinx",
 ]
 
 # This points to aboutcode.readthedocs.io
 # In case of "undefined label" ERRORS check docs on intersphinx to troubleshoot
 # Link was created at commit - https://github.com/nexB/aboutcode/commit/faea9fcf3248f8f198844fe34d43833224ac4a83
 
 intersphinx_mapping = {
-    'aboutcode': ('https://aboutcode.readthedocs.io/en/latest/', None),
-    'scancode-workbench': ('https://scancode-workbench.readthedocs.io/en/develop/', None),
+    "aboutcode": ("https://aboutcode.readthedocs.io/en/latest/", None),
+    "scancode-workbench": ("https://scancode-workbench.readthedocs.io/en/develop/", None),
 }
 
 
@@ -62,7 +62,7 @@
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ["_static"]
 
-master_doc = 'index'
+master_doc = "index"
 
 html_context = {
     "display_github": True,
@@ -72,9 +72,7 @@
     "conf_py_path": "/docs/source/",  # path in the checkout to the docs root
 }
 
-html_css_files = [
-    '_static/theme_overrides.css'
-    ]
+html_css_files = ["_static/theme_overrides.css"]
 
 
 # If true, "Created using Sphinx" is shown in the HTML footer. Default is True.