Merge 337_enhance_check_command

 * Add an option to fetch license from ScanCode LicneseDB
 * Enhance `check` command to validate `license_expression` from
ScanCode LicenseDB or DJC

Signed-off-by: Chin Yeung Li <[email protected]>

Author: chinyeungli

CHANGELOG.rst

@@ -1,5 +1,9 @@
 2021-xx-xx
+<<<<<<< HEAD
     Release 7.0.0
+=======
+    Release x.x.x
+>>>>>>> refs/heads/337_enhance_check_command
 
     * Add '@' as a support character for filename #451
     * Add support to collect redistributable sources #22
@@ -11,6 +15,7 @@
     * Update configuration scripts
     * Use readthedocs for documentation
     * Add Dockerfile to run aboutcode with docker
+    * Add new option to choose extract license from ScanCode LicenseDB or DJC License Library 
 
 2021-04-02
     Release 6.0.0

docs/source/general.rst

@@ -79,10 +79,10 @@ You should start with a software inventory of your codebase in spreadsheet or JS
       - Optional. You can separate each identifier using " OR " and " AND " to document the relationship between multiple license identifiers, such as a choice among multiple licenses.
     * - license_key
       - ScanCode license key for the component.
-      - Optional. gen will obtain license information from DejaCode Enterprise if the --fetch-license option is set, including the license text, in order to create and write the appropriate .LICENSE file in the .ABOUT file target directory.
+      - Optional. gen will obtain license information from ScanCode LicenseDB or DejaCode Enterprise if the --fetch-license or --fetch-license-djc option is set, including the license text, in order to create and write the appropriate .LICENSE file in the .ABOUT file target directory.
     * - license_name
       - License name for the component.
-      - Optional. This field will be generated if the --fetch-license option is set.
+      - Optional. This field will be generated if the --fetch-license or --fetch-license-djc option is set.
     * - license file
       - license file name   
       - Optional. gen will look for the file name (if a directory is specified in the --reference option) to copy that file to the .ABOUT file target directory.
@@ -223,11 +223,11 @@ Here is an example of a gen command:
 
         ..  code-block:: none
 
-                about gen --fetch-license {{your license library api key}} --reference /Users/harrypotter/myAboutFiles/ /Users/harrypotter/myAboutFiles/myProject-bom.csv /Users/harrypotter/myAboutFiles/
+                about gen --fetch-license --reference /Users/harrypotter/myAboutFiles/ /Users/harrypotter/myAboutFiles/myProject-bom.csv /Users/harrypotter/myAboutFiles/
 
 This gen example command does the following:
 
--   Activates the --fetch-license option to get license text.
+-   Activates the --fetch-license option to get license information.
 
 -   Activates the --reference option to get license text files and notice text files that you have specified in your software inventory to be copied next to the associated .ABOUT files when those are created.
 

docs/source/reference.rst

@@ -134,6 +134,8 @@ Options
 
         ..  code-block:: none
 
+                --djc api_url api_key  Validate license_expression from a DejaCode License
+                                         Library API URL using the API KEY.
                 --verbose                Show all the errors and warning
                 -h, --help               Show this message and exit.
 
@@ -154,6 +156,11 @@ Details
                 
                 $ about check --verbose /home/project/about_files/
 
+Special Notes
+-------------
+If no `--djc` option is set, the tool will default to check license_expression from
+ScanCode LicenseDB.
+
 collect_redist_src
 ==================
 
@@ -249,19 +256,20 @@ Options
                 --android                           Generate MODULE_LICENSE_XXX (XXX will be
                                                     replaced by license key) and NOTICE as the same
                                                     design as from Android.
-                
-                --fetch-license api_url api_key     Fetch licenses data from DejaCode License
+                --fetch-license                     Fetch license data and text files from the
+                                                    ScanCode LicenseDB.
+                --fetch-license-djc api_url api_key Fetch licenses data from DejaCode License
                                                     Library and create <license>.LICENSE
                                                     side-by-side with the generated .ABOUT file.
                                                     The following additional options are required:
-                
+
                                                     api_url - URL to the DejaCode License Library
                                                     API endpoint
-                
+
                                                     api_key - DejaCode API key
                                                     Example syntax:
-                
-                                                    about gen --fetch-license 'api_url' 'api_key'
+
+                                                    about gen --fetch-license-djc api_url api_key
                 --reference PATH                    Path to a directory with reference license
                                                     data and text files.
                 -q, --quiet                         Do not print any error/warning.
@@ -279,44 +287,53 @@ Details
         ..  code-block:: none
 
                 --android
-                
+
                     Create an empty file named `MODULE_LICENSE_XXX` where `XXX` is the license
                     key and create a NOTICE file which these two files follow the design from
                     Android Open Source Project.
-                
+
                     The input **must** have the license key information as this is needed to
                     create the empty MODULE_LICENSE_XXX
-                
+
                 $ about gen --android LOCATION OUTPUT
-                
+
                 --fetch-license
-                
+
+                    Fetch licenses text and create <license>.LICENSE side-by-side
+                    with the generated .ABOUT file using the data fetched from the the ScanCode LicenseDB.
+
+                    The input needs to have the 'license_expression' field.
+
+                $ about gen --fetch-license LOCATION OUTPUT
+
+                --fetch-license-djc
+
                     Fetch licenses text from a DejaCode API, and create <license>.LICENSE side-by-side
                     with the generated .ABOUT file using the data fetched from the DejaCode License Library.
-                
+
                     This option requires 2 parameters:
                         api_url - URL to the DJE License Library.
                         api_key - Hash key to authenticate yourself in the API.
-                
+
                     In addition, the input needs to have the 'license_expression' field.
                     (Please contact nexB to get the api_* value for this feature)
-                
-                $ about gen --fetch-license 'api_url' 'api_key' LOCATION OUTPUT
-                
+
+                $ about gen --fetch-license-djc 'api_url' 'api_key' LOCATION OUTPUT
+
                 --reference
-                
+
                     Copy the reference files such as 'license_files' and 'notice_files' to the
                     generated location from the specified directory.
-                
+
                     For instance,
                     the specified directory, /home/licenses_notices/, contains all the licenses and notices:
                     /home/licenses_notices/apache2.LICENSE
                     /home/licenses_notices/jquery.js.NOTICE
-                
+
                 $ about gen --reference /home/licenses_notices/ LOCATION OUTPUT
-                
+
                 --verbose
-                
+
                     This option tells the tool to show all errors found.
                     The default behavior will only show 'CRITICAL', 'ERROR', and 'WARNING'
 

src/attributecode/cmd.py

@@ -35,6 +35,7 @@
 from attributecode.gen import generate as generate_about_files, load_inventory
 from attributecode.model import collect_inventory, get_copy_list
 from attributecode.model import copy_redist_src
+from attributecode.model import pre_process_and_fetch_license_dict
 from attributecode.model import write_output
 from attributecode.util import extract_zip
 from attributecode.util import filter_errors
@@ -210,9 +211,13 @@ def inventory(location, output, format, quiet, verbose):  # NOQA
 
 # FIXME: the CLI UX should be improved with two separate options for API key and URL
 @click.option('--fetch-license',
+    is_flag=True,
+    help='Fetch license data and text files from the ScanCode LicenseDB.')
+
+@click.option('--fetch-license-djc',
     nargs=2,
     type=str,
-    metavar='URL KEY',
+    metavar='api_url api_key',
     help='Fetch license data and text files from a DejaCode License Library '
          'API URL using the API KEY.')
 
@@ -230,7 +235,7 @@ def inventory(location, output, format, quiet, verbose):  # NOQA
     help='Show all error and warning messages.')
 
 @click.help_option('-h', '--help')
-def gen(location, output, android, fetch_license, reference, quiet, verbose):
+def gen(location, output, android, fetch_license, fetch_license_djc, reference, quiet, verbose):
     """
 Given a CSV/JSON inventory, generate ABOUT files in the output location.
 
@@ -252,6 +257,7 @@ def gen(location, output, android, fetch_license, reference, quiet, verbose):
         android=android,
         reference_dir=reference,
         fetch_license=fetch_license,
+        fetch_license_djc=fetch_license_djc,
     )
 
     errors = unique(errors)
@@ -469,20 +475,40 @@ def collect_redist_src(location, output, from_inventory, with_structures, zip, q
     type=click.Path(
         exists=True, file_okay=True, dir_okay=True, readable=True, resolve_path=True))
 
+@click.option('--djc',
+    nargs=2,
+    type=str,
+    metavar='api_url api_key',
+    help='Validate license_expression from a DejaCode License Library '
+         'API URL using the API KEY.')
+
 @click.option('--verbose',
     is_flag=True,
     help='Show all error and warning messages.')
 
 @click.help_option('-h', '--help')
-def check(location, verbose):
+def check(location, djc, verbose):
     """
 Check .ABOUT file(s) at LOCATION for validity and print error messages.
 
 LOCATION: Path to an ABOUT file or a directory with ABOUT files.
     """
     print_version()
+    api_url = ''
+    api_key = ''
+    if djc:
+        # Strip the ' and " for api_url, and api_key from input
+        api_url = djc[0].strip("'").strip('"')
+        api_key = djc[1].strip("'").strip('"')
     click.echo('Checking ABOUT files...')
-    errors, _abouts = collect_inventory(location)
+    errors, abouts = collect_inventory(location)
+
+
+    # Validate license_expression
+    key_text_dict, errs = pre_process_and_fetch_license_dict(abouts, api_url, api_key)
+    for e in errs:
+        errors.append(e)
+
     errors = unique(errors)
     severe_errors_count = report_errors(errors, quiet=False, verbose=verbose)
     sys.exit(severe_errors_count)

src/attributecode/gen.py

@@ -222,7 +222,7 @@ def update_about_resource(self):
     pass
 
 
-def generate(location, base_dir, android=None, reference_dir=None, fetch_license=False):
+def generate(location, base_dir, android=None, reference_dir=None, fetch_license=False, fetch_license_djc=False):
     """
     Load ABOUT data from a CSV inventory at `location`. Write ABOUT files to
     base_dir. Return errors and about objects.
@@ -234,10 +234,13 @@ def generate(location, base_dir, android=None, reference_dir=None, fetch_license
     gen_license = False
     # FIXME: use two different arguments: key and url
     # Check if the fetch_license contains valid argument
-    if fetch_license:
+    if fetch_license_djc:
         # Strip the ' and " for api_url, and api_key from input
-        api_url = fetch_license[0].strip("'").strip('"')
-        api_key = fetch_license[1].strip("'").strip('"')
+        api_url = fetch_license_djc[0].strip("'").strip('"')
+        api_key = fetch_license_djc[1].strip("'").strip('"')
+        gen_license = True
+
+    if fetch_license:
         gen_license = True
 
     # TODO: WHY use posix??

src/attributecode/model.py

@@ -30,6 +30,7 @@
 import posixpath
 import traceback
 from itertools import zip_longest
+import urllib
 from urllib.parse import urljoin
 from urllib.parse import urlparse
 from urllib.request import urlopen
@@ -1523,48 +1524,69 @@ def save_as_csv(location, about_dicts, field_names):
 def pre_process_and_fetch_license_dict(abouts, api_url, api_key):
     """
     Modify a list of About data dictionaries by adding license information
-    fetched from the DejaCode API.
+    fetched from the ScanCode LicenseDB or DejaCode API.
     """
-    dje_uri = urlparse(api_url)
-    domain = '{uri.scheme}://{uri.netloc}/'.format(uri=dje_uri)
-    dje_lic_urn = urljoin(domain, 'urn/?urn=urn:dje:license:')
     key_text_dict = {}
     captured_license = []
     errors = []
+    if api_url:
+        dje_uri = urlparse(api_url)
+        domain = '{uri.scheme}://{uri.netloc}/'.format(uri=dje_uri)
+        lic_urn = urljoin(domain, 'urn/?urn=urn:dje:license:')
+        url = api_url
+    else:
+        url = 'https://scancode-licensedb.aboutcode.org/'
     if util.have_network_connection():
-        if not valid_api_url(api_url):
-            msg = u"URL not reachable. Invalid '--api_url'. License generation is skipped."
+        if not valid_api_url(url):
+            msg = u"URL not reachable. Invalid 'URL'. License generation is skipped."
             errors.append(Error(ERROR, msg))
     else:
         msg = u'Network problem. Please check your Internet connection. License generation is skipped.'
         errors.append(Error(ERROR, msg))
+
+    if errors:
+        return key_text_dict, errors
+    
     for about in abouts:
-        # No need to go through all the about objects for license extraction if we detected
-        # invalid '--api_key'
+        # No need to go through all the about objects if '--api_key' is invalid
         auth_error = Error(ERROR, u"Authorization denied. Invalid '--api_key'. License generation is skipped.")
         if auth_error in errors:
             break
         if about.license_expression.present:
             special_char_in_expression, lic_list = parse_license_expression(about.license_expression.value)
             if special_char_in_expression:
-                msg = (u"The following character(s) cannot be in the license_expression: " +
+                msg = (about.about_file_path + u": The following character(s) cannot be in the license_expression: " +
                        str(special_char_in_expression))
                 errors.append(Error(ERROR, msg))
             else:
                 for lic_key in lic_list:
                     if not lic_key in captured_license:
+                        lic_url = ''
+                        license_text = ''
                         detail_list = []
-                        license_name, license_key, license_text, errs = api.get_license_details_from_api(api_url, api_key, lic_key)
-                        for e in errs:
-                            if e not in errors:
-                                errors.append(e)
-                        if license_key:
-                            captured_license.append(lic_key)
-                            dje_lic_url = dje_lic_urn + license_key
-                            detail_list.append(license_name)
-                            detail_list.append(license_text)
-                            detail_list.append(dje_lic_url)
-                            key_text_dict[license_key] = detail_list
+                        if api_key:
+                            license_name, _license_key, license_text, errs = api.get_license_details_from_api(url, api_key, lic_key)
+                            for severity, message in errs: 
+                                msg = (about.about_file_path + ": " + message)
+                                errors.append(Error(severity, msg))
+                            lic_url = lic_urn + lic_key
+                        else:
+                            license_url = url + lic_key + '.json'
+                            license_text_url = url + lic_key + '.LICENSE'
+                            try:
+                                json_url = urlopen(license_url)
+                                data = json.loads(json_url.read())
+                                license_name = data['name']
+                                license_text = urllib.request.urlopen(license_text_url).read().decode('utf-8')
+                                lic_url = url + data['key'] + '.LICENSE'
+                            except:
+                                msg = about.about_file_path + u" : Invalid 'license': " + lic_key
+                                errors.append(Error(ERROR, msg))
+                        captured_license.append(lic_key)
+                        detail_list.append(license_name)
+                        detail_list.append(license_text)
+                        detail_list.append(lic_url)
+                        key_text_dict[lic_key] = detail_list
     return key_text_dict, errors
 
 
@@ -1595,6 +1617,7 @@ def valid_api_url(api_url):
         # This will always goes to exception as no key are provided.
         # The purpose of this code is to validate the provided api_url is correct
         urlopen(request)
+        return True
     except HTTPError as http_e:
         # The 403 error code is refer to "Authentication credentials were not provided.".
         # This is correct as no key are provided.

tests/testdata/test_cmd/help/about_check_help.txt

@@ -5,5 +5,7 @@ Usage: about check [OPTIONS] LOCATION
   LOCATION: Path to an ABOUT file or a directory with ABOUT files.
 
 Options:
-  --verbose   Show all error and warning messages.
-  -h, --help  Show this message and exit.
+  --djc api_url api_key  Validate license_expression from a DejaCode License
+                         Library API URL using the API KEY.
+  --verbose              Show all error and warning messages.
+  -h, --help             Show this message and exit.