Simplify controls.json format (#63)

Author: alfinkel

CHANGES.md

@@ -1,5 +1,8 @@
-# UNRELEASED
+# 1.3.0
 
+- [IMPROVED] Simplified `controls.json` format.  Original format is also supported.
+- [NEW] Added documentation for `controls.json` and check execution.
+- [NEW] Added ControlDescriptor unit tests.
 - [FIXED] ComplianceFetcher session object is auto-closed now in tearDownClass.
 
 # 1.2.7

compliance/__init__.py

@@ -14,4 +14,4 @@
 # limitations under the License.
 """Compliance automation package."""
 
-__version__ = '1.2.7'
+__version__ = '1.3.0'

compliance/controls.py

@@ -30,15 +30,14 @@ class ControlDescriptor(object):
 
     def __init__(self, dirs=None):
         """Construct and initialize the ControlDescriptor object."""
-        dirs = dirs or [os.path.abspath('.')]
         self._controls = {}
         self._paths = []
-        for d in dirs:
-            json_file = os.path.join(d, 'controls.json')
+        for d in dirs or ['.']:
+            json_file = os.path.join(os.path.abspath(d), 'controls.json')
             if not os.path.isfile(json_file):
                 continue
             self._controls.update(json.loads(open(json_file).read()))
-        self._paths.append(json_file)
+            self._paths.append(json_file)
 
     @property
     def paths(self):
@@ -54,24 +53,30 @@ def as_dict(self):
     def accred_checks(self):
         """Provide all checks by accreditation (key) as a dictionary."""
         if not hasattr(self, '_accred_checks'):
-            self._accred_checks = defaultdict(list)
-            for check, evidence in self._controls.items():
-                for control in evidence.values():
-                    for accreds in control.values():
-                        for accred in accreds:
-                            self._accred_checks[accred].append(check)
+            self._accred_checks = defaultdict(set)
+            for check in self._controls.keys():
+                accreds = self.get_accreditations(check)
+                for accred in accreds:
+                    self._accred_checks[accred].add(check)
         return self._accred_checks
 
     def get_accreditations(self, test_path):
         """
         Provide the accreditation list for a given test_path.
 
-        :param test_path: the Python path to the test. For instance:
-          ``package.accr1.TestClass`` or ``package.accr2.test_function``
+        This works for original and simplified controls formats.
+
+        Original: {'test_path': {'key': {'sub-key: ['accred1', 'accred2']}}}
+        Simplified: {'test_path': ['accred1', 'accred2']}
+
+        :param test_path: the Python path to the test.
+          For example: ``package.category.checks.module.TestClass``
         """
-        test_paths = self._controls.get(test_path, {})
+        check_details = self._controls.get(test_path, {})
+        if isinstance(check_details, list):
+            return set(check_details)
         accreditations = [
-            itertools.chain(*c.values()) for c in test_paths.values()
+            itertools.chain(*c.values()) for c in check_details.values()
         ]
         return set(itertools.chain(*accreditations))
 

doc-source/design-principles.rst

@@ -22,6 +22,7 @@ The tool is divided in two main parts:
 Both fetch and check phases are run by unittest. This is very convenient
 as fetchers and checks are loaded automatically by ``unittest``.
 
+
 Evidences
 ~~~~~~~~~
 
@@ -145,7 +146,7 @@ This is a list of modifications that are completely forbidden:
 * Adding live-generated data that does not come from the source.
 
 * Applying `check-like` logic (e.g. your data update if it includes an
-  `if`). Checkers should test the evidence, not fetchers.
+  `if`). Checks should test the evidence, not fetchers.
 
 Evidence Validation
 ===================
@@ -231,6 +232,13 @@ due to an unavailable evidence dependency.
       ...
       return json.dumps(foo_bar_data)
 
+Fetcher Execution
+=================
+
+The Auditree framework will run all fetchers (tests prefixed by ``fetch_``)
+that it can find.
+
+
 Compliance Checks
 ~~~~~~~~~~~~~~~~~
 
@@ -242,8 +250,8 @@ provided on the command line.
 Checks *assume* that all evidence is retrieved by fetchers.  Consequently
 checks **should not** be used to retrieve or store any ``RawEvidence`` in the
 evidence locker. Each check class may have from one to multiple checks defined
-(that is, a check is a method prefixed with `test_` in a check class). Each of
-these checks will be executed by the compliance tool with the following
+(that is, a check is a method prefixed with ``test_`` in a check class). Each of
+these checks will be executed by the Auditree framework with the following
 possible results:
 
 * ``OK``: the check ran successfully and **passed** all validations.
@@ -359,12 +367,34 @@ prior to executing the check's logic.
               self.add_warnings('bar stuff', warnings)
               self.add_successes('bar stuff', successes)
 
+Check Execution
+===============
+
+The Auditree framework executes checks (tests prefixed by ``test_``) based
+on accreditation groupings defined in a ``controls.json`` config file.
+This is especially useful when targeting check result content to the
+appropriate groups of people.  The framework will by default look for
+``controls.json`` in the current directory.  It is possible to supply the
+framework with alternate ``controls.json`` location(s) by providing an
+alternate path or paths at the end of a compliance check execution command via
+the CLI.  In the case of multiple locations, the framework will combine the
+content of all ``controls.json`` files found together.  With this check to
+accreditation mapping, the framework can execute checks based on the
+accreditations passed to the framework by the CLI.
+
+``controls.json`` content format example::
+
+  {
+    "chk_pkg.chk_cat_foo.checks.chk_module_foo.FooCheckClass": ["accred.one"],
+    "chk_pkg.chk_cat_bar.checks.chk_module_bar.BarCheckClass": ["accred.one", "accred.two"]
+  }
+
 
 Fixers
 ~~~~~~
 
 After checks have been run, but before notifications or reports are
-generated, the compliance tool will optionally try to fix the
+generated, the Auditree framework will optionally try to fix the
 issues automatically. This is controlled with the ``--fix`` option.
 By default it is ``off``, and this is the mode that is used during the
 daily CI runs in Travis. But you can also set it to ``dry-run`` or ``on``.
@@ -379,6 +409,7 @@ in the notification message.
 
 See :ref:`fixers` section for more information.
 
+
 Report Builder
 ~~~~~~~~~~~~~~
 
@@ -402,10 +433,11 @@ messages to stdout, etc).
 
 See :ref:`notifiers-description` section for more information.
 
+
 Execution Config
 ~~~~~~~~~~~~~~~~
 
-The compliance tool is designed to be run locally from your PC or from
+The Auditree framework is designed to be run locally from your PC or from
 a CI server like Jenkins or Travis. The execution can be tweaked at 2
 levels:
 
@@ -422,6 +454,7 @@ levels:
 
 .. _credentials:
 
+
 Credentials
 ~~~~~~~~~~~
 

doc-source/quick-start.rst

@@ -9,7 +9,7 @@ There might be some terms in this quick start that you don't fully
 understand, so you may want to have a look into
 :ref:`design-principles` section.
 
-The compliance tool requires, at least, one directory with the
+The Auditree framework requires, at least, one directory with the
 fetchers and checks. In order to learn how to use it, let's use a
 demo folder with a simple set of fetchers and checks. You can find it
 at `doc/demo-checks`::
@@ -59,7 +59,7 @@ mode::
 
   $ compliance --fetch --evidence no-push -C setup.json .
 
-The compliance tool will ``pull`` the repository to
+The Auditree framework will ``pull`` the repository to
 ``/tmp/compliance`` (only if it does not exist already), then run all
 the fetchers that need to be run (i.e. those whose evidence TTL
 has expired) but it will **not** push anything. This is handy for
@@ -97,10 +97,17 @@ instead::
 controls.json
 -------------
 
-The mapping between accreditations and checks happens at
-``controls.json``. New checks must be included here in order to be
-collected by the compliance tool. Note that this is not the same case
-as in fetchers, where all of them are executed.
+The mapping between a check (check class path) and accreditations
+(a list of accreditations) happens in the ``controls.json`` config
+file.  Each new check must be included in ``controls.json`` in
+order to be considered for execution by the Auditree framework.
+Note that this is not the case for fetchers, where all are executed.
+As an example, the format for ``controls.json`` is as follows::
+
+  {
+    "chk_pkg.chk_cat_foo.checks.chk_module_foo.FooCheckClass": ["accred.one"],
+    "chk_pkg.chk_cat_bar.checks.chk_module_bar.BarCheckClass": ["accred.one", "accred.two"]
+  }
 
 
 A few recommendations

doc-source/report-builder.rst

@@ -5,7 +5,7 @@
 Report Builder
 ==============
 
-After tests have been run by the compliance tool, the
+After tests have been run by the Auditree framework, the
 :py:class:`~compliance.report.ReportBuilder` is executed in order to
 render all the required reports. This is how it works:
 
@@ -43,7 +43,7 @@ render all the required reports. This is how it works:
    notify that the error occurred, in standard output. Note that report
    generation will not halt on an error, and will attempt to generate all other
    reports.
-   
+
 
 .. _templating:
 

doc-source/running-on-travis.rst

@@ -5,7 +5,7 @@
 Running on Travis
 =================
 
-Running compliance tool from a CI like Travis can be really useful for
+Running the Auditree framework from a CI like Travis can be really useful for
 executing your compliance checks periodically. Thus you can track the
 current level of compliance for different standards and also notify
 people whenever there is a failure, so it can be fixed in some way.

test/fixtures/controls/original/controls.json

@@ -0,0 +1,7 @@
+{
+  "foo_pkg.checks.test_foo_module.FooCheck": {
+    "foo_evidence": {
+      "foo_control": ["accred.foo"]
+    }
+  }
+}

test/fixtures/controls/simplified/controls.json

@@ -0,0 +1 @@
+{"bar_pkg.checks.test_bar_module.BarCheck": ["accred.bar"]}

test/t_compliance/t_controls/__init__.py

@@ -0,0 +1,15 @@
+# -*- mode:python; coding:utf-8 -*-
+# Copyright (c) 2020 IBM Corp. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Compliance automation control descriptor tests module."""