Merge branch 'develop' into get-valid-yaml

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Author: AyanSinhaMahapatra

AUTHORS.rst

@@ -79,6 +79,7 @@ The following organizations or individuals have contributed to ScanCode:
 - Sebastian Schuberth @sschuberth
 - Shankhadeep Dey @Iamshankhadeep
 - Sharikzama @ZamaSharik
+- Shrijal Acharya @OctoPie23
 - Shivam Chauhan @chashiv
 - Shivam Sandbhor @sbs2001
 - Steven Esser @majurg

CHANGELOG.rst

@@ -63,6 +63,12 @@ Package detection:
   manifest data parsing code outside of the scancode-toolkit context in other
   libraries.
 
+- The PackageData model now includes a ``holder`` field, which is populated with
+  holder data extracted from the copyright field if copyright data is present,
+  otherwise it remains empty.
+
+  https://github.com/nexB/scancode-toolkit/issues/3290
+
 
 License detection:
 ~~~~~~~~~~~~~~~~~~~

CONTRIBUTING.rst

@@ -3,7 +3,7 @@ Contributing
 ============
 
 Contributions are welcome and appreciated!
-Every little bit helps, and credit will always be given.
+Every little bit helps, and a credit will always be given.
 
 .. _issues : https://github.com/nexB/scancode-toolkit/issues
 __ issues_
@@ -14,15 +14,15 @@ check `easy issues <https://github.com/nexB/scancode-toolkit/labels/easy>`_
 When contributing to ScanCode (such as code, bugs, documentation, etc.) you
 agree to the Developer `Certificate of Origin <http://developercertificate.org/>`_
 and the ScanCode license (see the `NOTICE <https://github.com/nexB/scancode-toolkit/blob/develop/NOTICE>`_ file).
-The same approach is used by the Linux Kernel developers and several other projects.
+The same approach is used by Linux Kernel developers and several other projects.
 
 For commits, it is best to simply add a line like this to your commit message,
 with your name and email::
 
     Signed-off-by: Jane Doe <[email protected]>
 
 Please try to write a good commit message, see `good commit message wiki
-<https://aboutcode.readthedocs.io/en/latest/contributing/writing_good_commit_messages.html>` for
+<https://aboutcode.readthedocs.io/en/latest/contributing/writing_good_commit_messages.html>`_ for
 details. In particular use the imperative for your commit subject: think that
 you are giving an order to the codebase to update itself.
 
@@ -35,7 +35,7 @@ To send feedback or ask a question, `file an issue <issues_>`_
 If you are proposing a feature:
 
 * Explain how it would work.
-* Keep the scope simple possible to make it easier to implement.
+* Keep the scope as simple as possible to make it easier to implement.
 * Remember that your contributions are welcomed to implement this feature!
 
 
@@ -54,23 +54,21 @@ For other questions, discussions, and chats, we have:
   This is a busy place with a lot of CI and commit notifications that makes
   actual chat sometimes difficult!
 
-- a mailing list at `sourceforge <https://lists.sourceforge.net/lists/listinfo/aboutcode-discuss>`_
-
 - a Gitter channel to discuss Documentation at https://gitter.im/aboutcode-org/gsod-season-of-docs
 
 Bug reports
 ===========
 
 When `reporting a bug`__ please include:
 
-* Your operating system name, version and architecture (32 or 64 bits).
+* Your operating system name, version, and architecture (32 or 64 bits).
 * Your Python version.
 * Your ScanCode version.
 * Any additional details about your local setup that might be helpful to
   diagnose this bug.
 * Detailed steps to reproduce the bug, such as the commands you ran and a link
   to the code you are scanning.
-* The errors messages or failure trace if any.
+* The error messages or failure trace if any.
 * If helpful, you can add a screenshot as an issue attachment when relevant or
   some extra file as a link to a `Gist <https://gist.github.com>`_.
 
@@ -83,11 +81,11 @@ any other general upgrades, etc. Even a minor typo fix is welcomed.
 
 If something is missing in the documentation or if you found some part confusing,
 please file an issue with your suggestions for improvement. Use the “Documentation Improvement”
-template. Your help and contributions makes ScanCode docs better, we love hearing from you!
+template. Your help and contribution make ScanCode docs better, we love hearing from you!
 
 The ScanCode documentation is hosted at `scancode-toolkit.readthedocs.io <https://scancode-toolkit.readthedocs.io/en/latest/>`_.
 
-If you want to contribute to Scancode Dcoumentation, you'll find `this guide here <https://scancode-toolkit.readthedocs.io/en/latest/contribute/contrib_doc.html>`_ helpful.
+If you want to contribute to Scancode Documentation, you'll find `this guide here <https://scancode-toolkit.readthedocs.io/en/latest/contribute/contrib_doc.html>`_ helpful.
 
 Development
 ===========
@@ -124,11 +122,13 @@ To set up ScanCode for local development:
 3. Create a branch for local development::
 
     git checkout -b name-of-your-bugfix-or-feature
+    
+4. Check out the Contributing to Code Development `documentation <https://scancode-toolkit.readthedocs.io/en/stable/contribute/contrib_dev.html>`_, as it contains more in-depth guide for contributing code and documentation.
 
-4. To configure your local environment for development, locate to the main
-   directory of the local repository, run the configure script.
+5. To configure your local environment for development, locate to the main
+   directory of the local repository, and run the configure script.
    The configure script creates an isolated Python `virtual environment` in
-   your checkout directory, the Python `pip` tool, and installs the thirdparty
+   your checkout directory, the Python `pip` tool, and installs the third-party
    libraries (from the `thirdparty/ directory`), setup the paths, etc.
    See https://virtualenv.pypa.io/en/latest/ for more details. 
 
@@ -157,10 +157,20 @@ To set up ScanCode for local development:
    When you pull new code from git, rerun ./configure
 
 
-5. Now you can make your code changes in your local clone.
+6. Now you can make your code changes in your local clone.
    Please create new unit tests for your code. We love tests!
 
-6. When you are done with your changes, run all the tests.
+7. An update to the ``CHANGELOG`` is required if any important changes are made that needs to be communicated such as:
+
+   * Changes in the API.
+
+   * Addition or deletion of CLI options.
+
+   * Addition of any new feature or any other miscellaneous changes to the program.
+   
+8. If there is a code change, a significant document, or any other changes, you must update the ``AUTHORS`` to include your own name.
+
+9. When you are done with your changes, run all the tests.
    Use this command::
 
         py.test
@@ -169,7 +179,7 @@ To set up ScanCode for local development:
 
        py.test -n6
 
-   If you are running this on a RedHat based OS you may come across this
+   If you are running this on a RedHat-based OS you may come across this
    failure::
 
        OSError: libbz2.so.1.0: cannot open shared object file: No such file or directory
@@ -182,18 +192,18 @@ To set up ScanCode for local development:
 
    See `this issue <https://github.com/nexB/scancode-toolkit/issues/443>`_ for more information.
 
-7. Check the status of your local repository before commit, regarding files changed::
+10. Check the status of your local repository before committing, regarding files changed::
 
-    git status
+     git status
 
 
-8. Commit your changes and push your branch to your GitHub fork::
+11. Commit your changes and push your branch to your GitHub fork::
 
-    git add <file-changed-1> <file-changed-2> <file-changed-3>
-    git commit -m "Your detailed description of your changes." --signoff
-    git push <repository-alias-name> name-of-your-bugfix-or-feature
+     git add <file-changed-1> <file-changed-2> <file-changed-3>
+     git commit -m "Your detailed description of your changes." --signoff
+     git push <repository-alias-name> name-of-your-bugfix-or-feature
 
-9. Submit a pull request through the GitHub website for this branch.
+12. Submit a pull request through the GitHub website for this branch.
 
 
 Pull Request Guidelines
@@ -205,7 +215,7 @@ create a pull request. You can add new commits to your branch as needed.
 For merging, your request would need to:
 
 1. Include unit tests that are passing (run ``py.test``).
-2. Update documentation as needed for new API, functionality etc.
+2. Update documentation as needed for new API, functionality, etc.
 3. Add a note to ``CHANGELOG.rst`` about the changes.
 4. Add your name to ``AUTHORS.rst``.
 
@@ -228,3 +238,4 @@ To run tests in parallel on eight processors::
 To run tests verbosely, displaying all print statements to terminal::
 
     py.test  -vvs
+

azure-pipelines.yml

@@ -91,15 +91,6 @@ jobs:
                 venv/bin/pytest -vvs --test-suite=plugins \
                   tests/licensedcode/test_additional_license.py
 
-    - template: etc/ci/azure-posix.yml
-      parameters:
-          job_name: ubuntu18_cpython
-          image_name: ubuntu-18.04
-          python_versions: ['3.8', '3.9', '3.10', '3.11']
-          python_architecture: x64
-          test_suites:
-              all: venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py --reruns 2
-
     - template: etc/ci/azure-posix.yml
       parameters:
           job_name: ubuntu20_cpython
@@ -147,9 +138,9 @@ jobs:
 
     - template: etc/ci/azure-win.yml
       parameters:
-          job_name: win2019_cpython_2
-          image_name: windows-2019
-          python_versions: ['3.9', '3.10', '3.11']
+          job_name: win2022_cpython
+          image_name: windows-2022
+          python_versions: ['3.8', '3.9', '3.10', '3.11']
           python_architecture: x64
           test_suites:
               all: venv\Scripts\pytest -n 2 -vvs tests\scancode\test_cli.py --reruns 2
@@ -206,14 +197,6 @@ jobs:
           test_suites:
               all: venv/bin/pip install --upgrade-strategy eager --force-reinstall --upgrade -e .[testing] && venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py
 
-    - template: etc/ci/azure-posix.yml
-      parameters:
-          job_name: ubuntu18_cpython_latest_from_pip
-          image_name: ubuntu-18.04
-          python_versions: ['3.8', '3.9', '3.10', '3.11']
-          test_suites:
-              all: venv/bin/pip install --upgrade-strategy eager --force-reinstall --upgrade -e .[testing] && venv/bin/pytest -n 2 -vvs tests/scancode/test_cli.py
-
     - template: etc/ci/azure-posix.yml
       parameters:
           job_name: ubuntu20_cpython_latest_from_pip

configure

@@ -159,7 +159,7 @@ PROD_REQUIREMENTS="scancode_toolkit*.whl"
 VIRTUALENV_DIR=venv
 
 # Cleanable files and directories to delete with the --clean option
-CLEANABLE="build venv"
+CLEANABLE="build dist venv .cache .eggs"
 
 # extra  arguments passed to pip
 PIP_EXTRA_ARGS=" "
@@ -175,6 +175,13 @@ CFG_ROOT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
 CFG_BIN_DIR=$CFG_ROOT_DIR/$VIRTUALENV_DIR/bin
 
 
+################################
+# Install with or without and index. With "--no-index" this is using only local wheels
+# This is an offline mode with no index and no network operations
+# NO_INDEX="--no-index "
+NO_INDEX=""
+
+
 ################################
 # Thirdparty package locations and index handling
 # Find packages from the local thirdparty directory if present
@@ -308,6 +315,7 @@ while getopts :-: optchar; do
     esac
 done
 
+
 PIP_EXTRA_ARGS="$PIP_EXTRA_ARGS"
 
 find_python

configure.bat

@@ -35,7 +35,7 @@ set "REL_REQUIREMENTS=--requirement etc/scripts/requirements.txt"
 set "VIRTUALENV_DIR=venv"
 
 @rem # Cleanable files and directories to delete with the --clean option
-set "CLEANABLE=build venv"
+set "CLEANABLE=build dist venv .cache .eggs"
 
 @rem # extra  arguments passed to pip
 set "PIP_EXTRA_ARGS= "

docs/source/conf.py

@@ -28,16 +28,16 @@
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
-'sphinx.ext.intersphinx',
+    "sphinx.ext.intersphinx",
 ]
 
 # This points to aboutcode.readthedocs.io
 # In case of "undefined label" ERRORS check docs on intersphinx to troubleshoot
 # Link was created at commit - https://github.com/nexB/aboutcode/commit/faea9fcf3248f8f198844fe34d43833224ac4a83
 
 intersphinx_mapping = {
-    'aboutcode': ('https://aboutcode.readthedocs.io/en/latest/', None),
-    'scancode-workbench': ('https://scancode-workbench.readthedocs.io/en/develop/', None),
+    "aboutcode": ("https://aboutcode.readthedocs.io/en/latest/", None),
+    "scancode-workbench": ("https://scancode-workbench.readthedocs.io/en/develop/", None),
 }
 
 
@@ -62,7 +62,7 @@
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ["_static"]
 
-master_doc = 'index'
+master_doc = "index"
 
 html_context = {
     "display_github": True,
@@ -72,9 +72,7 @@
     "conf_py_path": "/docs/source/",  # path in the checkout to the docs root
 }
 
-html_css_files = [
-    '_static/theme_overrides.css'
-    ]
+html_css_files = ["_static/theme_overrides.css"]
 
 
 # If true, "Created using Sphinx" is shown in the HTML footer. Default is True.

etc/scripts/README.rst

@@ -21,7 +21,7 @@ Pre-requisites
     virtualenv or in the the main configured development virtualenv.
     These requireements need to be installed::
 
-        pip install --requirement etc/release/requirements.txt
+        pip install --requirement etc/scripts/requirements.txt
 
 TODO: we need to pin the versions of these tools
 
@@ -34,7 +34,7 @@ Scripts
 ~~~~~~~
 
 **gen_requirements.py**: create/update requirements files from currently
-  installed requirements. 
+  installed requirements.
 
 **gen_requirements_dev.py** does the same but can subtract the main requirements
   to get extra requirements used in only development.
@@ -50,7 +50,7 @@ The sequence of commands to run are:
 
     ./configure --clean
     ./configure
-    python etc/release/gen_requirements.py --site-packages-dir <path to site-packages dir>
+    python etc/scripts/gen_requirements.py --site-packages-dir <path to site-packages dir>
 
 * You can optionally install or update extra main requirements after the
   ./configure step such that these are included in the generated main requirements.
@@ -59,7 +59,7 @@ The sequence of commands to run are:
 
     ./configure --clean
     ./configure --dev
-    python etc/release/gen_requirements_dev.py --site-packages-dir <path to site-packages dir>
+    python etc/scripts/gen_requirements_dev.py --site-packages-dir <path to site-packages dir>
 
 * You can optionally install or update extra dev requirements after the
   ./configure step such that these are included in the generated dev