Fixed $480 - Ability to transform Excel formatted file

 * Enhance functionality of transform to work with .xlsx
 * Update docs and changelog

Signed-off-by: Chin Yeung Li <[email protected]>

Author: chinyeungli

CHANGELOG.rst

@@ -1,9 +1,5 @@
 2021-xx-xx
-<<<<<<< HEAD
     Release 7.0.0
-=======
-    Release x.x.x
->>>>>>> refs/heads/337_enhance_check_command
 
     * Add '@' as a support character for filename #451
     * Add support to collect redistributable sources #22
@@ -15,7 +11,8 @@
     * Update configuration scripts
     * Use readthedocs for documentation
     * Add Dockerfile to run aboutcode with docker
-    * Add new option to choose extract license from ScanCode LicenseDB or DJC License Library 
+    * Add new option to choose extract license from ScanCode LicenseDB or DJC License Library
+    * Add ability to transform Excel formatted file 
 
 2021-04-02
     Release 6.0.0

docs/source/general.rst

@@ -19,7 +19,7 @@ AboutCode Toolkit is a tool for your software development team to document your
 
 -   **inventory**: Generate a Software Inventory list (.csv or .json format) from your codebase based on ABOUT file(s). Note that this Software Inventory will only include components that have AboutCode Toolkit data. In another word, if you do not create AboutCode Toolkit files for your own original software components, these components will not show up in the generated inventory.
 
--   **transform**: A command to transform an input CSV/JSON by applying renaming and/or filtering and then output to a new CSV/JSON file.
+-   **transform**: A command to transform an input CSV/JSON/Excel by applying renaming and/or filtering and then output to a new CSV/JSON/Excel file.
 
 Additional AboutCode Toolkit information is available at:
 
@@ -168,11 +168,11 @@ Fields Renaming and Optional Custom Fields
 
 Since your input's field name may not match with the AboutCode Toolkit standard field name, you can use the transform subcommand to do the transformation.
 
-A transform configuration file is used to describe which transformations and validations to apply to a source CSV/JSON file. This is a simple text file using YAML format, using the same format as an .ABOUT file.
+A transform configuration file is used to describe which transformations and validations to apply to a source CSV/JSON/Excel file. This is a simple text file using YAML format, using the same format as an .ABOUT file.
 
 The attributes that can be set in a configuration file are:
 
--   field_renamings: An optional map of source field name to target new field name that is used to rename CSV/JSON fields.
+-   field_renamings: An optional map of source field name to target new field name that is used to rename CSV/JSON/Excel fields.
 
         ..  code-block:: none
 
@@ -184,7 +184,7 @@ The attributes that can be set in a configuration file are:
 The renaming is always applied first before other transforms and checks. All other field names referenced below are AFTER the renaming have been applied.
 For instance with this configuration, the field "Directory/Location" will be renamed to "about_resource" and "foo" to "bar":
 
--   required_fields: An optional list of required field names that must have a value, beyond the standard field names. If a source CSV/JSON does not have such a field or an entry is missing a value for a required field, an error is reported.
+-   required_fields: An optional list of required field names that must have a value, beyond the standard field names. If a source CSV/JSON/Excel does not have such a field or an entry is missing a value for a required field, an error is reported.
 
 For instance with this configuration, an error will be reported if the fields "name" and "version" are missing, or if any entry does not have a value set for these fields:
 

docs/source/reference.rst

@@ -34,7 +34,7 @@ Commands
                 gen                 Generate .ABOUT files from an inventory as CSV or JSON.
                 inventory           Collect the inventory of .ABOUT files to a CSV or JSON
                                     file.
-                transform           Transform a CSV/JSON by applying renamings, filters and checks.
+                transform           Transform a CSV/JSON/Excel by applying renamings, filters and checks.
 
 attrib
 ======
@@ -446,8 +446,8 @@ Syntax
 
                 about transform [OPTIONS] LOCATION OUTPUT
                 
-                LOCATION: Path to a CSV/JSON file.
-                OUTPUT: Path to CSV/JSON inventory file to create.
+                LOCATION: Path to a CSV/JSON/Excel file.
+                OUTPUT: Path to CSV/JSON/Excel inventory file to create.
 
 Options
 -------
@@ -464,7 +464,7 @@ Options
 Purpose
 -------
 
-Transform the CSV/JSON file at LOCATION by applying renamings, filters and checks and then write a new CSV/JSON to OUTPUT (Format for input and output need to be the same).
+Transform the CSV/JSON/Excel file at LOCATION by applying renamings, filters and checks and then write a new CSV/JSON/Excel to OUTPUT (Format for input and output need to be the same).
 
 Details
 ^^^^^^^

docs/source/specification.rst

@@ -116,7 +116,7 @@ Fields order and multiple occurrences
 
 The field order does not matter. Multiple occurrences of a field name is not supported.
 
-The tool processing an ABOUT file or CSV/JSON input will issue an error when a field name occurs more than once in the input file.
+The tool processing an ABOUT file or CSV/JSON/Excel input will issue an error when a field name occurs more than once in the input file.
 
 Field referencing a file
 ------------------------

src/attributecode/cmd.py

@@ -37,6 +37,10 @@
 from attributecode.model import copy_redist_src
 from attributecode.model import pre_process_and_fetch_license_dict
 from attributecode.model import write_output
+from attributecode.transform import transform_csv_to_csv
+from attributecode.transform import transform_json_to_json
+from attributecode.transform import transform_excel_to_excel
+from attributecode.transform import Transformer
 from attributecode.util import extract_zip
 from attributecode.util import filter_errors
 from attributecode.util import get_temp_dir
@@ -527,17 +531,17 @@ def print_config_help(ctx, param, value):
 
 
 @about.command(cls=AboutCommand,
-    short_help='Transform a CSV/JSON by applying renamings, filters and checks.')
+    short_help='Transform a CSV/JSON/Excel by applying renamings, filters and checks.')
 
 @click.argument('location',
     required=True,
-    callback=partial(validate_extensions, extensions=('.csv', '.json',)),
+    callback=partial(validate_extensions, extensions=('.csv', '.json', '.xlsx',)),
     metavar='LOCATION',
     type=click.Path(exists=True, dir_okay=False, readable=True, resolve_path=True))
 
 @click.argument('output',
     required=True,
-    callback=partial(validate_extensions, extensions=('.csv', '.json',)),
+    callback=partial(validate_extensions, extensions=('.csv', '.json', '.xlsx',)),
     metavar='OUTPUT',
     type=click.Path(exists=False, dir_okay=False, writable=True, resolve_path=True))
 
@@ -563,18 +567,14 @@ def print_config_help(ctx, param, value):
 @click.help_option('-h', '--help')
 def transform(location, output, configuration, quiet, verbose):  # NOQA
     """
-Transform the CSV/JSON file at LOCATION by applying renamings, filters and checks
-and then write a new CSV/JSON to OUTPUT (Format for input and output need to be
+Transform the CSV/JSON/Excel file at LOCATION by applying renamings, filters and checks
+and then write a new CSV/JSON/Excel to OUTPUT (Format for input and output need to be
 the same).
 
-LOCATION: Path to a CSV/JSON file.
+LOCATION: Path to a CSV/JSON/Excel file.
 
-OUTPUT: Path to CSV/JSON inventory file to create.
+OUTPUT: Path to CSV/JSON/Excel inventory file to create.
     """
-    from attributecode.transform import transform_csv_to_csv
-    from attributecode.transform import transform_json_to_json
-    from attributecode.transform import Transformer
-
     if not configuration:
         transformer = Transformer.default()
     else:
@@ -584,6 +584,8 @@ def transform(location, output, configuration, quiet, verbose):  # NOQA
         errors = transform_csv_to_csv(location, output, transformer)
     elif location.endswith('.json') and output.endswith('.json'):
         errors = transform_json_to_json(location, output, transformer)
+    elif location.endswith('.xlsx') and output.endswith('.xlsx'):
+        errors = transform_excel_to_excel(location, output, transformer)
     else:
         msg = 'Extension for the input and output need to be the same.'
         click.echo(msg)

src/attributecode/transform.py

@@ -15,10 +15,12 @@
 
 import io
 import json
-from collections import Counter
+from collections import Counter, OrderedDict
 from itertools import zip_longest
 
 import attr
+import itertools
+import openpyxl
 
 from attributecode import CRITICAL
 from attributecode import Error
@@ -48,7 +50,7 @@ def transform_csv_to_csv(location, output, transformer):
         msg = u'Duplicated field name: %(name)s'
         for name in dupes:
             errors.append(Error(CRITICAL, msg % locals()))
-        return field_names, [], errors
+        return errors
 
     # Convert to dicts
     new_data = [dict(zip_longest(field_names, item)) for item in data]
@@ -84,6 +86,31 @@ def transform_json_to_json(location, output, transformer):
         return []
 
 
+def transform_excel_to_excel(location, output, transformer):
+    """
+    Read a Excel file at `location` and write a new Excel file at `output`. Apply
+    transformations using the `transformer` Transformer.
+    Return a list of Error objects.
+    """
+    if not transformer:
+        raise ValueError('Cannot transform without Transformer')
+
+    dupes, new_data = read_excel(location)
+    errors = []
+    if dupes:
+        msg = u'Duplicated field name: %(name)s'
+        for name in dupes:
+            errors.append(Error(CRITICAL, msg % locals()))
+        return errors
+
+    _field_names, updated_data, errors = transform_data(new_data, transformer)
+    if errors:
+        return errors
+    else:
+        write_excel(output, updated_data)
+        return []
+
+
 def strip_trailing_fields_csv(names):
     """
     Strip trailing spaces for field names #456
@@ -385,3 +412,53 @@ def write_json(location, data):
     """
     with open(location, 'w') as jsonfile:
         json.dump(data, jsonfile, indent=3)
+
+def read_excel(location):
+    """
+    Read Excel at `location`, return a list of ordered dictionaries, one
+    for each row.
+    """
+    results = []
+    errors = []
+    sheet_obj = openpyxl.load_workbook(location).active
+    max_col = sheet_obj.max_column
+
+    index = 1
+    col_keys = []
+    mapping_dict = {}
+    while index <= max_col:
+        value = sheet_obj.cell(row=1, column=index).value
+        if value in col_keys:
+            msg = 'Duplicated column name, ' + str(value) + ', detected.' 
+            errors.append(Error(CRITICAL, msg))
+            return errors, results
+        if value in mapping_dict:
+            value = mapping_dict[value]
+        col_keys.append(value)
+        index = index + 1
+
+    for row in sheet_obj.iter_rows(min_row=2, values_only=True):
+        row_dict = OrderedDict()
+        index = 0
+        while index < max_col:
+            value = row[index]
+            if value:
+                row_dict[col_keys[index]] = value
+            else:
+                row_dict[col_keys[index]] = ''
+            index = index + 1
+        results.append(row_dict)
+    return errors, results
+
+
+def write_excel(location, data):
+    wb = openpyxl.Workbook()
+    ws = wb.active
+
+    headers = list(set(itertools.chain.from_iterable(data)))
+    ws.append(headers)
+
+    for elements in data:
+        ws.append([elements.get(h) for h in headers])
+
+    wb.save(location)

tests/test_cmd.py

@@ -347,7 +347,8 @@ def check_about_stdout(options, expected_loc, regen=False):
     expected_file = get_test_loc(expected_loc, must_exists=True)
     with open(expected_file, 'r') as ef:
         expected = ef.read()
-
+    print("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
+    print(result.output)
     assert expected.splitlines(False) == result.output.splitlines(False)
 
 

tests/testdata/test_cmd/help/about_help.txt

@@ -20,5 +20,5 @@ Commands:
   gen                 Generate .ABOUT files from an inventory as CSV or JSON.
   inventory           Collect the inventory of .ABOUT files to a CSV or JSON
                       file.
-  transform           Transform a CSV/JSON by applying renamings, filters and
-                      checks.
+  transform           Transform a CSV/JSON/Excel by applying renamings, filters
+                      and checks.

tests/testdata/test_cmd/help/about_transform_help.txt

@@ -1,17 +1,17 @@
 Usage: about transform [OPTIONS] LOCATION OUTPUT
 
-  Transform the CSV/JSON file at LOCATION by applying renamings, filters and
-  checks and then write a new CSV/JSON to OUTPUT (Format for input and output
-  need to be the same).
+  Transform the CSV/JSON/Excel file at LOCATION by applying renamings, filters
+  and checks and then write a new CSV/JSON/Excel to OUTPUT (Format for input and
+  output need to be the same).
 
-  LOCATION: Path to a CSV/JSON file.
+  LOCATION: Path to a CSV/JSON/Excel file.
 
-  OUTPUT: Path to CSV/JSON inventory file to create.
+  OUTPUT: Path to CSV/JSON/Excel inventory file to create.
 
 Options:
   -c, --configuration FILE  Path to an optional YAML configuration file. See
                             --help-format for format help.
   --help-format             Show configuration file format help and exit.
   -q, --quiet               Do not print error or warning messages.
   --verbose                 Show all error and warning messages.
-  -h, --help                Show this message and exit.
+  -h, --help                Show this message and exit.