Merge branch 'develop' into 2110-dart-pubspec

Author: pombredanne

CHANGELOG.rst

@@ -4,8 +4,12 @@ Changelog
 v21.x.x (next, future)
 -----------------------
 
-Breaking API changes:
-~~~~~~~~~~~~~~~~~~~~~
+Important API changes:
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+ - The data structure of the JSON output is now versioned and the next version
+   is available with a new command line option. We are also documenting a new
+   and clear API policy and backward compatibility policy.
 
  - The data structure of the JSON output has changed for copyrights, authors
    and holders: we now use proper name for attributes and not a generic "value".
@@ -23,6 +27,83 @@ Breaking API changes:
    that contains each package instance that can be aggregating data from
    multiple manifests for a single package instance.
 
+ - The data structure for HTML output has been changed to include emails and urls under the 
+   "infos" object. Now HTML template will output holders, authors, emails, and 
+   urls into separate tables like "licenses" and "copyrights".
+
+Copyright detection:
+~~~~~~~~~~~~~~~~~~~~
+
+ - The data structure in the JSON is now using consistently named attributes as
+   opposed to a plain value.
+
+
+Package detection:
+~~~~~~~~~~~~~~~~~~
+
+ - Add support for OpenWRT packages.
+ - Add support for Yocto/BitBake .bb recipes.
+ - Add support to track installed files for each Package type.
+
+
+License detection:
+~~~~~~~~~~~~~~~~~~~
+
+- Unknown licenses have a new flag "is_unknown" to identify them
+  beyond just the naming convention of having "unknown" as part of their name.
+
+- Rules that match at least one unknown license have a flag "has_unknown" set
+  in the returned match results.
+
+
+Many thanks to every contributors that made this possible and in particular:
+
+- Akanksha Garg @akugarg
+- Ayan Sinha Mahapatra @AyanSinhaMahapatra
+- Jono Yang @JonoYang
+- Philippe Ombredanne @pombredanne
+
+
+
+v21.8.4
+---------
+
+This is a minor bug fix release primarily for Windows installation.
+There is no feature change.
+
+Installation:
+~~~~~~~~~~~~~~~~~~
+
+- Application installation on Windows works again. This fixes #2610
+- We now build and test app bundles on all supported Python versions: 3.6 to 3.9
+
+
+Thank you to @gunaztar for reporting the #2610 bug
+
+Documentation:
+~~~~~~~~~~~~~~~~~~
+
+- Documentation is updated to reference supported Python versions 3.6 to 3.9
+
+
+
+v21.7.30
+---------
+
+This is a minor release with several bug fixes, major performance improvements
+and support for new and improved package formats
+
+
+Many thanks to every contributors that made this possible and in particular:
+
+- Abhigya Verma @abhi27-web
+- Ayan Sinha Mahapatra @AyanSinhaMahapatra
+- Dennis Clark @DennisClark
+- Jono Yang @JonoYang
+- Mayur Agarwal @mrmayurgithub 
+- Philippe Ombredanne @pombredanne
+- Pierre Tardy @tardyp
+
 
 Outputs:
 ~~~~~~~~
@@ -31,28 +112,35 @@ Outputs:
    the JSON output
  - Add new Debian machine readable copyright output.
  - The CSV output "Resource" column has been renamed to "path".
- - The SPDX output now has the mandatory DocumentNamespace attribut per SPDX specs #2344
+ - The SPDX output now has the mandatory DocumentNamespace attribute per SPDX specs #2344
 
 
 Copyright detection:
 ~~~~~~~~~~~~~~~~~~~~
 
- - The data structure in the JSON is now using consistently named attributes as
-   opposed to a plain value.
-
  - The copyright detection speed has been significantly improved with the tests
    taking roughly 1/2 of the time to run. This is achieved mostly by replacing
    NLTK with a the minimal and simplified subset we need in a new library named
    pygmars.
 
+License detection:
+~~~~~~~~~~~~~~~~~~~
+
+ - Add new licenses: now tracking 1763 licenses
+ - Add new license detection rules: now tracking 29475 license detection rules
+ - We have also improved license expression parsing and processing
+
 
 Package detection:
 ~~~~~~~~~~~~~~~~~~
 
- - Add support for OpenWRT packages.
- - Add support for Yocto/BitBake .bb recipes.
+ - The Debian packages declared license detection has been significantly improved.
  - The Alpine packages declared license detection has been significantly improved.
- - Add support to track installed files for each Package type.
+ - There is new support for shell parsing and Alpine packages APKBUILD data collection.
+ - There is new support for various Windows packages detection using multiple
+   techniques including MSI, Windows registry and several more.
+ - There is new support for Distroless Debian-like installed packages.
+ - There is new support for Dart Pub package manifests.
 
 
 v21.6.7

CONTRIBUTING.rst

@@ -134,25 +134,34 @@ To set up ScanCode for local development:
 
    Run this command to configure ScanCode::
 
-        ./configure
+        ./configure --dev
 
    On Windows use instead::
 
-        configure 
+        configure --dev
 
-   Then run this: `source bin/activate` or `. bin/activate` 
-   (or run `bin\\activate` on Windows)
+   Then activate the virtual environment::
+
+        source bin/activate
+
+        or
+
+        . bin/activate
+
+   On Windows use::
+
+        Scripts\activate
 
    When you create a new terminal/shell to work on ScanCode rerun the activate step.
 
    When you pull new code from git, rerun ./configure
-   
+
 
 5. Now you can make your code changes in your local clone.
    Please create new unit tests for your code. We love tests!
 
 6. When you are done with your changes, run all the tests.
-   Use this command:: 
+   Use this command::
 
         py.test
 
@@ -162,25 +171,25 @@ To set up ScanCode for local development:
 
    If you are running this on a RedHat based OS you may come across this
    failure::
-   
+
        OSError: libbz2.so.1.0: cannot open shared object file: No such file or directory
- 
+
    Try creating a symbolic link to libbz2.so.1.0 to solve this issue::
 
        locate libbz2.so.1.0
        cd <resulting libbz2.so directory>
        sudo ln -s <your version of libbz2.so> libbz2.so.1.0
-      
+
    See `this issue <https://github.com/nexB/scancode-toolkit/issues/443>`_ for more information.
 
 7. Check the status of your local repository before commit, regarding files changed::
-    
+
     git status
 
 
 8. Commit your changes and push your branch to your GitHub fork::
 
-    git add <file-changed-1> <file-changed-2> <file-changed-3> 
+    git add <file-changed-1> <file-changed-2> <file-changed-3>
     git commit -m "Your detailed description of your changes." --signoff
     git push <repository-alias-name> name-of-your-bugfix-or-feature
 
@@ -196,7 +205,7 @@ create a pull request. You can add new commits to your branch as needed.
 For merging, your request would need to:
 
 1. Include unit tests that are passing (run ``py.test``).
-2. Update documentation as needed for new API, functionality etc. 
+2. Update documentation as needed for new API, functionality etc.
 3. Add a note to ``CHANGELOG.rst`` about the changes.
 4. Add your name to ``AUTHORS.rst``.
 

INSTALL.rst

@@ -16,13 +16,12 @@ Prerequisites
 -------------
 
 Before installing ScanCode make sure you have  installed these prerequisites.
-The main one is a Python interpreter.
-Python 3.6 is required for the standard installation.
+The main one is to have Python installed version 3.6, 3.7, 3.8 or 3.9.
 
 - For Linux(Ubuntu): ``sudo apt install python3.6-dev bzip2 xz-utils zlib1g libxml2-dev libxslt1-dev``
-- For MacOS: Install Python 3.6.8 from https://www.python.org/ftp/python/3.6.8/python-3.6.8-macosx10.9.pkg
-- For Windows: Install Python 3.6.8 from https://www.python.org/ftp/python/3.6.8/python-3.6.8-amd64.exe
-- For FreeBSD: (this ineeds to be documented)
+- For MacOS: Install Python 3.x from https://www.python.org/
+- For Windows: Install Python 3.x from https://www.python.org/  using the 64 bits amd64 variant
+- For FreeBSD: (this needs to be documented)
 
 Refer `Prerequisites <https://scancode-toolkit.readthedocs.io/en/latest/getting-started/install.html#prerequisites>`_
 for detailed information on all different operating systems and Python versions.
@@ -46,9 +45,9 @@ Use a release download and install as an application
 Advanced installation: pip install a Python PyPI package
 --------------------------------------------------------
 
-- Create a virtual environment for Python 3.6 (of higher) and activate it::
+- Create a virtual environment for Python 3.x and activate it::
 
-    virtualenv -p /usr/bin/python3.6 venv-scancode && source venv-scancode/bin/activate
+    virtualenv -p python3 venv-scancode && source venv-scancode/bin/activate
 
 - Run ``pip install scancode-toolkit[full]``
 

README.rst

@@ -120,8 +120,7 @@ Installation
 ============
 
 Before installing ScanCode make sure that you have installed the prerequisites
-properly. This means installing Python (Python 3.6 or higher is required.
-When installing the app tarball or zip, only Python 3.6 is supported for now).
+properly. This means installing Python (Python 3.6 or higher is required).
 
 See `prerequisites <https://scancode-toolkit.readthedocs.io/en/latest/getting-started/install.html#prerequisites>`_
 for detailed information on the support platforms and Python versions.

configure

@@ -45,10 +45,12 @@ CLEANABLE="
     .eggs
     pip-selfcheck.json
     src/scancode_toolkit.egg-info
+    src/scancode_toolkit_mini.egg-info
     SCANCODE_DEV_MODE
     man
     Scripts"
 
+
 # extra  arguments passed to pip
 PIP_EXTRA_ARGS=" "
 
@@ -65,12 +67,17 @@ CFG_BIN_DIR=$CFG_ROOT_DIR/$VIRTUALENV_DIR/bin
 
 ################################
 # scancode-specific: Thirdparty package locations and index handling
-# Do we have thirdparty/files? use the mit.LICENSE as a proxy
-if [[ -f "$CFG_ROOT_DIR/thirdparty/mit.LICENSE" ]]; then
-    LINKS=$CFG_ROOT_DIR/thirdparty
+if [[ "$PYPI_LINKS" == "" ]]; then
+    # Do we have thirdparty/files? use the mit.LICENSE as a proxy
+    if [[ -f "$CFG_ROOT_DIR/thirdparty/mit.LICENSE" ]]; then
+        LINKS=$CFG_ROOT_DIR/thirdparty
+    else
+        LINKS=https://thirdparty.aboutcode.org/pypi
+    fi
 else
-    LINKS=https://thirdparty.aboutcode.org/pypi
+    LINKS=$PYPI_LINKS
 fi
+
 PIP_EXTRA_ARGS="--no-index --find-links $LINKS"
 ################################
 

docs/source/cli-reference/output-format.rst

@@ -29,8 +29,6 @@ following options.
     others. Scancode Workbench and other applications that use Scancode Result data as input accept
     only the ``json`` format.
 
-    .. include::  /rst_snippets/note_snippets/output_json_notdef.rst
-
     The following code performs a scan on the samples directory, and publishes the results in
     ``json`` format::
 
@@ -179,6 +177,7 @@ following options.
               "short_name": "MIT Old Style",
               "category": "Permissive",
               "is_exception": false,
+              "is_unknown": false,
               "owner": "MIT",
               "homepage_url": "http://fedoraproject.org/wiki/Licensing:MIT#Old_Style",
               "text_url": "http://fedoraproject.org/wiki/Licensing:MIT#Old_Style",

docs/source/cli-reference/synopsis.rst

@@ -228,6 +228,7 @@ A sample JSON output for an individual file will look like::
           "short_name": "MIT Old Style",
           "category": "Permissive",
           "is_exception": false,
+          "is_unknown": false,
           "owner": "MIT",
           "homepage_url": "http://fedoraproject.org/wiki/Licensing:MIT#Old_Style",
           "text_url": "http://fedoraproject.org/wiki/Licensing:MIT#Old_Style",

docs/source/conf.py

@@ -18,8 +18,8 @@
 # -- Project information -----------------------------------------------------
 
 project = 'ScanCode-Toolkit'
-copyright = '2019 ScanCode-Toolkit.org'
-author = 'ScanCode-Toolkit.org'
+copyright = 'nexB Inc. and others.'
+author = 'AboutCode.org authors and contributors'
 
 
 # -- General configuration ---------------------------------------------------