Update RTD and doc

chinyeungli · chinyeungli · commit 58dd243cf0b5 · 2021-11-10T17:02:33.000+08:00
Signed-off-by: Chin Yeung Li &lt;tli@nexb.com&gt;
diff --git a/docs/source/general.rst b/docs/source/general.rst
@@ -9,15 +9,15 @@ AboutCode Toolkit Defined
 
 AboutCode Toolkit is a tool for your software development team to document your code inside your codebase, typically in preparation for a product release, side-by-side with the actual code. ABOUT file(s) have a simple, standard format that identifies components and their associated licenses. The current AboutCode Toolkit subcommands are:
 
--   **attrib**: Generate a Product Attribution notice document (HTML format) from your ABOUT file(s). You can also generate documents for other purposes (such as a License Reference) by varying your input control file and your template.
+-   **attrib**: Generate a Product Attribution notice document from your ABOUT file(s), JSON, CSV or Excel. You can also generate documents for other purposes (such as a License Reference) by varying your input control file and your template.
 
 -   **check**: A simple command to validate the ABOUT file(s) and output errors/warnings if any on the terminal.
 
 -   **collect_redist_src**: A command to collect and copy sources that have 'redistribute' flagged as 'True' in ABOUT file(s) or from an inventory.
 
--   **gen**: Create ABOUT file(s) from a Software Inventory file (.csv or .json format) which is typically created from a software audit, and insert these AboutCode Toolkit files into your codebase. You can regenerate the AboutCode Toolkit files from a new Software Inventory file whenever you make changes.
+-   **gen**: Create ABOUT file(s) from a Software Inventory file (.csv, .json or .xlsx format) which is typically created from a software audit, and insert these AboutCode Toolkit files into your codebase. You can regenerate the AboutCode Toolkit files from a new Software Inventory file whenever you make changes.
 
--   **inventory**: Generate a Software Inventory list (.csv or .json format) from your codebase based on ABOUT file(s). Note that this Software Inventory will only include components that have AboutCode Toolkit data. In another word, if you do not create AboutCode Toolkit files for your own original software components, these components will not show up in the generated inventory.
+-   **inventory**: Generate a Software Inventory list (.csv, .json or .xlsx format) from your codebase based on ABOUT file(s). Note that this Software Inventory will only include components that have AboutCode Toolkit data. In another word, if you do not create AboutCode Toolkit files for your own original software components, these components will not show up in the generated inventory.
 
 -   **transform**: A command to transform an input CSV/JSON/Excel by applying renaming and/or filtering and then output to a new CSV/JSON/Excel file.
 
@@ -217,7 +217,7 @@ For instance with this configuration, the target file will not contain the "type
 Run gen to Generate ABOUT file(s)
 ---------------------------------
 
-When your software inventory is ready, you can save it as a .csv or .json file, and use it as input to run gen to generate ABOUT file(s). The official gen parameters are defined here: :ref:`reference`
+When your software inventory is ready, you can save it as a .csv, .json or .xlsx file, and use it as input to run gen to generate ABOUT file(s). The official gen parameters are defined here: :ref:`reference`
 
 Here is an example of a gen command:
 
@@ -227,7 +227,7 @@ Here is an example of a gen command:
 
 This gen example command does the following:
 
--   Activates the --fetch-license option to get license information.
+-   Activates the --fetch-license option to get license information from ScanCode LicenseDB.
 
 -   Activates the --reference option to get license text files and notice text files that you have specified in your software inventory to be copied next to the associated .ABOUT files when those are created.
 
@@ -247,8 +247,9 @@ Review the generated ABOUT file(s) to determine if it meets your requirements. H
                 license_expression: gpl-2.0
                 licenses:
                     -   key: gpl-2.0
-                        name: GNU General Public License 2.0
+                        name: GPL 2.0
                         file: gpl-2.0.LICENSE
+                        url: https://scancode-licensedb.aboutcode.org/gpl-2.0.LICENSE
                 owner: Red Hat
                 redistribute: Y
 
@@ -257,21 +258,6 @@ You can make appropriate changes to your input software inventory and then run g
 Using attrib to Generate a Product Attribution Notice Package
 =============================================================
 
-Prepare a Filtered Product BOM to Use as Input to attrib
---------------------------------------------------------
-
-The Software Inventory that you prepared for gen most likely includes components that do not need to appear in a product attribution notice package; for example:
-
--   Components in your codebase that are not Deployed on the final product (e.g. build tools, testing tools, internal documentation).
-
--   Components in your codebase under licenses that do not require attribution (e.g. proprietary packages, commercial products).
-
-There are two options here:
-
--   Edit the jinja2 template to only include the one that have value in attribute field such as: ``{% if about_object.attribute.value %}``
-
--   You should prepare a filtered version of your software inventory (the one that you used for gen) by removing the rows that identify components which should not be included in a product attribution notice package, and saving that filtered version as your Product BOM.
-
 Prepare an Attribution Template to Use as Input to attrib
 ---------------------------------------------------------
 
@@ -401,7 +387,7 @@ One of the major features of the ABOUT File specification is that the .ABOUT fil
 
 If your organization adopts the practice of manually creating and maintaining ABOUT file(s), you can easily re-create your software inventory from your codebase using inventory. The official inventory parameters are defined here: :ref:`reference`
 
-A successful execution of inventory will create a complete software inventory in .csv format or .json format based on defined format.
+A successful execution of inventory will create a complete software inventory in .csv, .json or .xlsx format based on defined format.
 
 
 
diff --git a/docs/source/reference.rst b/docs/source/reference.rst
@@ -27,13 +27,12 @@ Commands
 
         ..  code-block:: none
 
-                attrib              Generate an attribution document from .ABOUT files.
+                attrib              Generate an attribution document from JSON/CSV/Excel/.ABOUT files.
                 check               Validate that the format of .ABOUT files is correct and
                                     report errors and warnings.
-                collect_redist_src  Collect redistributable sources.
-                gen                 Generate .ABOUT files from an inventory as CSV or JSON.
-                inventory           Collect the inventory of .ABOUT files to a CSV or JSON
-                                    file.
+                collect-redist-src  Collect redistributable sources.
+                gen                 Generate .ABOUT files from an inventory as CSV/JSON/Excel.
+                inventory           Collect the inventory of .ABOUT files to a CSV/JSON/Excel file.
                 transform           Transform a CSV/JSON/Excel by applying renamings, filters and checks.
 
 attrib
@@ -46,51 +45,92 @@ Syntax
 
                 about attrib [OPTIONS] LOCATION OUTPUT
 
-                LOCATION: Path to a file, directory or .zip archive containing .ABOUT
-                files.
-                
+                INPUT: Path to a file (.ABOUT/.csv/.json/.xlsx), directory or .zip archive containing .ABOUT files.
+
                 OUTPUT: Path where to write the attribution document.
 
 Options
 -------
 
         ..  code-block:: none
 
-                --template FILE          Path to an optional custom attribution template to
-                                         generate the attribution document. If not provided
-                                         the default built-in template is used.
-                --vartext <key>=<value>  Add variable text as key=value for use in a custom
-                                         attribution template.
-                -q, --quiet              Do not print error or warning messages.
-                --verbose                Show all error and warning messages.
-                -h, --help               Show this message and exit.
+                --api_url URL                URL to DejaCode License Library.
+                --api_key KEY                API Key for the  DejaCode License Library
+                --min-license-score INTEGER  Attribute components that have license score
+                                             higher than the defined --min-license-score.
+                --scancode                   Indicate the input JSON file is from
+                                             scancode toolkit.
+                --reference DIR              Path to a directory with reference files where
+                                             "license_file" and/or "notice_file" located.
+                --template FILE              Path to an optional custom attribution template to
+                                             generate the attribution document. If not provided
+                                             the default built-in template is used.
+                --vartext <key>=<value>      Add variable text as key=value for use in a custom
+                                             attribution template.
+                -q, --quiet                  Do not print error or warning messages.
+                --verbose                    Show all error and warning messages.
+                -h, --help                   Show this message and exit.
 
 Purpose
 -------
 
-Generate an attribution file which contains the all license information from the LOCATION along with the license text.
+Generate an attribution file which contains license information from the INPUT along with the license text.
 
 Assume the following:
 
         ..  code-block:: none
 
-            '/home/about_files/'** contains all the ABOUT files [LOCATION]
+            '/home/about_files/' contains all the ABOUT files [INPUT]
+            '/home/project/inventory.csv' is a BOM inventory [INPUT]
+            '/home/project/scancode-detection.json' is a detection output from scancode-toolkit[INPUT] 
+            '/home/project/licenses/' contains all the license/notice file references
             '/home/attribution/attribution.html' is the user's output path [OUTPUT]
 
+
+        ..  code-block:: none
+
             $ about attrib /home/about_files/ /home/attribution/attribution.html
+            or
+            $ about attrib /home/project/inventory.csv /home/attribution/attribution.html
+            or
+            $ about attrib --scancode /home/project/scancode-detection.json /home/attribution/attribution.html
 
 Details
 ^^^^^^^
 
         ..  code-block:: none
 
+                --api_url URL --api_key
+
+                    This option let user to define where to get the license information such as
+                    from DJE. If these options are not set, the tool will get the license
+                    information from ScanCode LicenseDB by default
+
+                $ about attrib --api_url <URL> --api_key <KEY> INPUT OUTPUT
+
+                --min-license-score
+
+                    This option is a filter to collect license information where the license score
+                    in the scancode toolkit detection is greater than or equal to the defined
+                    --min-license-score. This option is specifically design for scancode's input
+                    and therefore --scancode is required
+
+                $ about attrib --scancode --min-license-score 85 /home/project/scancode-detection.json OUTPUT
+
+                --reference
+
+                    This option is to define the reference directory where the 'license_file'
+                    or 'notice_file' are stored
+
+                $ about attrib --reference /home/project/licenses/ /home/project/inventory.csv OUTPUT
+
                 --template
-                
+
                     This option allows you to use your own template for attribution generation.
                     For instance, if you have a custom template located at:
                     /home/custom_template/template.html
                 
-                $ about attrib --template /home/custom_template/template.html LOCATION OUTPUT
+                $ about attrib --template /home/custom_template/template.html INPUT OUTPUT
                 
                 --vartext
                 
@@ -110,12 +150,7 @@ Details
 The following data are passed to jinja2 and, therefore, can be used for a custom template:
  * about object: the about objects
  * common_licenses: a common license keys list in licenses.py
- * license_file_key_and_context: a dictionary with license_file_key (It's basically a license_key if it's not a custom license or license file name otherwise) as a key and license text as the value
- * license_file_key_and_license_key: a dictionary with license file key as a key and license key as the value
- * license_file_name_and_license_file_key: a dictionary with license file name as a key and license file key as the value
- * license_key_and_license_file_name: a dictionary with license key as a key and license file name as the value
- * license_key_and_license_name: a dictionary with license key as a key and license name as the value
- * license_name_and_license_key: a dictionary with license name as a key and license key as the value
+ * licenses_list: a license object list contains all the licenses found in about objects. It contains the following attribute: key, name, filename, url, text
 
 check
 =====
@@ -245,7 +280,7 @@ Syntax
 
                 about gen [OPTIONS] LOCATION OUTPUT
                 
-                LOCATION: Path to a JSON or CSV inventory file.
+                LOCATION: Path to a JSON/CSV/Excel inventory file.
                 OUTPUT: Path to a directory where ABOUT files are generated.
 
 Options
@@ -279,7 +314,7 @@ Options
 Purpose
 -------
 
-Given a CSV/JSON inventory, generate ABOUT files in the output location.
+Given a CSV/JSON/Excel inventory, generate ABOUT files in the output location.
 
 Details
 ^^^^^^^
@@ -348,29 +383,29 @@ Syntax
                 about inventory [OPTIONS] LOCATION OUTPUT
                 
                 LOCATION: Path to an ABOUT file or a directory with ABOUT files.
-                OUTPUT: Path to the JSON or CSV inventory file to create.
+                OUTPUT: Path to the CSV/JSON/Excel inventory file to create.
 
 Options
 -------
 
         ..  code-block:: none
 
-                -f, --format [json|csv]     Set OUTPUT file format.  [default: csv]
-                -q, --quiet                 Do not print any error/warning.
-                --verbose                   Show all the errors and warning.
-                -h, --help                  Show this message and exit.
+                -f, --format [json|csv|excel]   Set OUTPUT file format.  [default: csv]
+                -q, --quiet                     Do not print any error/warning.
+                --verbose                       Show all the errors and warning.
+                -h, --help                      Show this message and exit.
 
 Purpose
 -------
 
-Create a JSON or CSV inventory of components from ABOUT files.
+Create a JSON/CSV/Excel inventory of components from ABOUT files.
 
 Details
 ^^^^^^^
 
         ..  code-block:: none
 
-                -f, --format [json|csv]
+                -f, --format [json|csv|excel]
                 
                     Set OUTPUT file format.  [default: csv]
                 
diff --git a/docs/source/specification.rst b/docs/source/specification.rst
@@ -26,8 +26,9 @@ A simple and valid ABOUT file named httpd-2.4.3.tar.gz.ABOUT may look like this:
                 license_expression: apache-2.0
                 licenses:
                     -   key: apache-2.0
-                        name: Apache License 2.0
+                        name: Apache 2.0
                         file: apache-2.0.LICENSE
+                        url: https://scancode-licensedb.aboutcode.org/apache-2.0.LICENSE
                 notice_file: httpd.NOTICE
                 copyright: Copyright (c) 2012 The Apache Software Foundation.
 
diff --git a/src/attributecode/cmd.py b/src/attributecode/cmd.py
@@ -324,7 +324,7 @@ def validate_template(ctx, param, value):
 
 @click.option('--min-license-score',
     type=int,
-    help='Attribute components that have license score higher than the defined '
+    help='Attribute components that have license score higher than or equal to the defined '
         '--min-license-score.')
 
 @click.option('--scancode',
@@ -363,7 +363,7 @@ def attrib(input, output, api_url, api_key, scancode, min_license_score, referen
     """
 Generate an attribution document at OUTPUT using JSON, CSV or Excel or .ABOUT files at INPUT.
 
-INPUT: Path to a file, directory or .zip archive containing .ABOUT files.
+INPUT: Path to a file (.ABOUT/.csv/.json/.xlsx), directory or .zip archive containing .ABOUT files.
 
 OUTPUT: Path where to write the attribution document.
     """
diff --git a/tests/testdata/test_cmd/help/about_attrib_help.txt b/tests/testdata/test_cmd/help/about_attrib_help.txt
@@ -3,15 +3,17 @@ Usage: about attrib [OPTIONS] INPUT OUTPUT
   Generate an attribution document at OUTPUT using JSON, CSV or Excel or .ABOUT
   files at INPUT.
 
-  INPUT: Path to a file, directory or .zip archive containing .ABOUT files.
+  INPUT: Path to a file (.ABOUT/.csv/.json/.xlsx), directory or .zip archive
+  containing .ABOUT files.
 
   OUTPUT: Path where to write the attribution document.
 
 Options:
   --api_url URL                URL to DejaCode License Library.
   --api_key KEY                API Key for the  DejaCode License Library
   --min-license-score INTEGER  Attribute components that have license score
-                               higher than the defined --min-license-score.
+                               higher than or equal to the defined --min-
+                               license-score.
   --scancode                   Indicate the input JSON file is from
                                scancode_toolkit.
   --reference DIR              Path to a directory with reference files where
