Return support for derived and temporary evidence (#123)

Author: alfinkel

.pre-commit-config.yaml

@@ -7,7 +7,7 @@ repos:
         files: "^(compliance|test)"
         stages: [commit]
 -   repo: https://gitlab.com/pycqa/flake8
-    rev: 3.9.1
+    rev: 3.9.2
     hooks:
     -   id: flake8
         args: [

CHANGES.md

@@ -1,3 +1,8 @@
+# [1.20.0](https://github.com/ComplianceAsCode/auditree-framework/releases/tag/v1.20.0)
+
+- [ADDED] Returned support for `DerivedEvidence`.
+- [ADDED] Returned support for `TmpEvidence`.
+
 # [1.19.1](https://github.com/ComplianceAsCode/auditree-framework/releases/tag/v1.19.1)
 
 - [FIXED] Evidence cache loading bug resolved.

compliance/__init__.py

@@ -14,4 +14,4 @@
 # limitations under the License.
 """Compliance automation package."""
 
-__version__ = '1.19.1'
+__version__ = '1.20.0'

compliance/evidence.py

@@ -31,8 +31,6 @@
 )
 from compliance.utils.path import FETCH_PREFIX, substitute_config
 
-from deprecated import deprecated
-
 HOUR = 60 * 60
 DAY = HOUR * 24
 YEAR = DAY * 365
@@ -236,7 +234,6 @@ def _partition(self, data, key):
         return data
 
 
-@deprecated(reason='Evidence class to be removed, use RawEvidence instead')
 class DerivedEvidence(_BaseEvidence):
     """The derived evidence class."""
 
@@ -255,7 +252,6 @@ def rootdir(self):
         return 'reports'
 
 
-@deprecated(reason='Evidence class to be removed')
 class TmpEvidence(_BaseEvidence):
     """The temporary evidence class."""
 
@@ -279,7 +275,7 @@ def rootdir(self):
 
 
 class _EvidenceContextManager(object):
-    """Base class for raw evidence context managers."""
+    """Base class for raw and temporary evidence context managers."""
 
     def __init__(self, locker, evidence_path, evidence_type):
         self.locker = locker
@@ -328,7 +324,6 @@ def __init__(self, locker, evidence_path):
         super().__init__(locker, evidence_path, 'raw')
 
 
-@deprecated(reason='Context manager to be removed')
 class tmp_evidence(_EvidenceContextManager):  # noqa: N801
     """
     Helper context manager for a typical ``fetch_`` method implementation.
@@ -357,7 +352,6 @@ def __init__(self, locker, evidence_path):
         super().__init__(locker, evidence_path, 'tmp')
 
 
-@deprecated(reason='Context manager to be removed, use raw_evidence instead')
 class derived_evidence(object):  # noqa: N801
     """
     Helper context manager for a typical ``fetch_`` method implementation.
@@ -573,7 +567,7 @@ def get_evidence_class(evidence_type):
     provided.  If no match ``None`` is returned.
 
     :param evidence_type: the type of evidence class desired as a string. Valid
-      values are ``reports``, ``raw``.
+      values are ``tmp``, ``reports``, ``derived``, ``raw``, ``external``.
 
     :returns: the appropriate evidence class.
     """
@@ -711,7 +705,6 @@ def wrapper(self, *args, **kwargs):
     return decorator
 
 
-@deprecated(reason='Decorator to be removed')
 def store_tmp_evidence(evidence_path):
     """
     Decorate a typical ``fetcher_`` method fetching temporary evidence.
@@ -735,14 +728,13 @@ def wrapper(self, *args, **kwargs):
     return decorator
 
 
-@deprecated(reason='Decorator to be removed, use store_raw_evidence instead')
 def store_derived_evidence(evidences, target):
     """
     Decorate a typical ``fetcher_`` method fetching derived evidence.
 
     Use when retrieving derived evidence by a fetcher and the name of the
     evidence is static/known.  When the evidence name is dynamic use the
-    tmp_evidence context manager instead.
+    derived_evidence context manager instead.
 
     The decorator expects that the decorated method returns the content to be
     stored in the locker.  The storing of the evidence is also handled by this
@@ -813,7 +805,6 @@ def decorator(f):
     return decorator
 
 
-@deprecated(reason='Decorator to be removed, use with_raw_evidences instead')
 def with_derived_evidences(*from_evidences):
     """
     Decorate a typical ``test_`` check method processing derived evidences.
@@ -833,7 +824,6 @@ def decorator(f):
     return decorator
 
 
-@deprecated(reason='Decorator to be removed')
 def with_tmp_evidences(*from_evidences):
     """
     Decorate a typical ``test_`` check method processing temporary evidences.

doc-source/design-principles.rst

@@ -26,7 +26,7 @@ as fetchers and checks are loaded automatically by ``unittest``.
 Evidence
 ~~~~~~~~
 
-Fetchers and checks manage evidence. We have defined three types of
+Fetchers and checks manage evidence. We have defined five types of
 evidence (see :py:mod:`compliance.evidence`):
 
 * :py:class:`~compliance.evidence.RawEvidence`: Gathered by
@@ -37,6 +37,18 @@ evidence (see :py:mod:`compliance.evidence`):
   as binary by setting the ``binary_content=True`` keyword argument key/value
   pair when constructing a ``RawEvidence`` object.
 
+* :py:class:`~compliance.evidence.DerivedEvidence`: Gathered/Generated by
+  fetchers and used by checks as *input*.  Derived evidence is useful for
+  those cases when a fetcher needs other evidence to perform computations
+  over data collected in order to generate a new evidence. This new
+  evidence is considered `derived` in the sense that its data is not the
+  same as the source.
+
+* :py:class:`~compliance.evidence.TmpEvidence`: Gathered by
+  fetchers and used by checks as *input*. This type of evidence is similar to
+  RawEvidence but it is never get pushed to the remote git repository. This is
+  useful for evidence that contains passwords or credentials.
+
 * :py:class:`~compliance.evidence.ExternalEvidence`: Planted in the locker
   with `plant <https://github.com/ComplianceAsCode/auditree-plant>`_
   and used by checks as *input*. For example, a list of users in GitHub.
@@ -212,7 +224,8 @@ A few examples of what it is allowed:
 
 In any case, any modification of a new raw evidence **must** be
 approved and agreed by the reviewers. By default, do **not** modify
-the raw data.
+the raw data.  If you need to, then you should consider using derived
+evidence.
 
 This is a list of modifications that are completely forbidden:
 
@@ -230,11 +243,12 @@ we've provided some helpful decorators and context managers that validate
 ``ttl`` for you and if necessary write the evidence to the evidence locker for
 you after it has been fetched.
 
-* ``store_raw_evidence`` decorator: Use this decorator on your fetcher method
-  when you know the full path and name of your raw evidence.  The decorator
-  takes as an argument, the path to your raw evidence as a string.
+* ``store_raw_evidence`` and ``store_tmp_evidence`` decorators: Use one of
+  these decorators on your fetcher method when you know the path and name of
+  your raw or tmp evidence.  The decorator takes as an argument, the path to
+  your raw or tmp evidence as a string.
 
-``@store_raw_evidence`` usage example::
+Usage example::
 
   ...
   from compliance.evidence import store_raw_evidence
@@ -248,15 +262,16 @@ you after it has been fetched.
       # The decorator will write it to the evidence locker
       return json.dumps(foo_bar_data)
 
-* ``raw_evidence`` context manager: Use this context manager within your
-  fetcher method when your fetcher retrieves multiple, similar raw evidence
-  based on a dynamic set of configurable values.  In other words the full name
-  and content of evidence is based on a configuration and not known prior
-  to execution of the fetcher logic.  The context manager takes as arguments, a
-  locker object and the path to your raw evidence as a string.  The context
-  manager yields the corresponding raw evidence object.
+* ``raw_evidence`` and ``tmp_evidence`` context managers: Use one of these
+  context managers within your fetcher method when your fetcher retrieves
+  multiple, similar raw or tmp evidence based on a dynamic set of configurable
+  values.  In other words the full name and content of evidence is based on a
+  configuration and not known prior to execution of the fetcher logic.  The
+  context manager takes as arguments, a locker object and the path to your raw
+  or tmp evidence as a string.  The context manager yields the corresponding
+  raw or tmp evidence object.
 
-``with raw_evidence`` usage example::
+Usage example::
 
   ...
   from compliance.evidence import raw_evidence
@@ -273,6 +288,109 @@ you after it has been fetched.
                   # Upon exit it is written to the evidence locker
                   evidence.set_content(json.dumps(foo_bar_data))
 
+* ``store_derived_evidence`` decorator: Use this decorator on your fetcher
+  method when you know the paths and names of your source evidences and
+  the path and name of your target derived evidence.  The decorator takes
+  as arguments, a list of source evidence paths as strings and a target derived
+  evidence path as a string.  It also passes the source evidences to the
+  decorated method in the form of method arguments.
+
+Usage example::
+
+  ...
+  from compliance.evidence import store_derived_evidence
+  ...
+  @store_derived_evidence(
+      ['raw/foo/evidence_bar.json', 'raw/foo/evidence_baz.json'],
+      'foo/derived_bar_baz.json'
+  )
+  fetch_foo_bar_baz_derived_evidence(self, bar_evidence, baz_evidence):
+      # Fetcher code only executes if evidence is stale
+      # Construct your derived evidence
+      derived_data = self._do_whatever(bar_evidence, baz_evidence)
+      # Return the content as a string
+      # The decorator will write it to the evidence locker
+      return json.dumps(derived_data)
+
+* ``derived_evidence`` context manager: Use this context manager within your
+  fetcher method when your fetcher generates multiple, similar derived
+  evidences based on a dynamic set of configurable values.  In other words the
+  name and content of the evidences are based on a configuration and not
+  known prior to execution of the fetcher logic.  The context manager takes as
+  arguments, a locker object, source evidence paths and a target derived
+  evidence path as a string.  The source evidence paths can be in the form of a
+  list of paths as strings, a dictionary of key/values pairs as strings where
+  the key is an evidence short name and the value is the evidence path, or
+  simply a single evidence path as a string.  The context manager yields a
+  dictionary containing the source and target evidences as the dictionary
+  values.  The source evidence key is its evidence path if a list of source
+  paths were provided or its evidence short name if a dictionary of paths were
+  provided or "source" if a single evidence path in the form of a string was
+  provided.  The target derived evidence key is always "derived".
+
+Usage example (source list provided)::
+
+  ...
+  from compliance.evidence import derived_evidence
+  ...
+  fetch_foo_bar_baz_derived_evidence(self):
+      for system in systems:
+          sources = ['raw/foo/evidence_bar.json', 'raw/foo/evidence_baz.json']
+          target = 'foo/derived_bar_baz_{}.json'.format(system)
+          with derived_evidence(self.locker, sources, target) as evidences:
+              # None is returned if target evidence is not stale
+              if evidences:
+                  # Construct your derived evidence
+                  derived_data = self._do_whatever(
+                      evidences['raw/foo/evidence_bar.json'],
+                      evidences['raw/foo/evidence_baz.json']
+                  )
+                  # Set the content as a string
+                  # Upon exit it is written to the evidence locker
+                  evidences['derived'].set_content(json.dumps(derived_data))
+
+Usage example (source dictionary provided)::
+
+  ...
+  from compliance.evidence import derived_evidence
+  ...
+  fetch_foo_bar_baz_derived_evidence(self):
+      for system in systems:
+          sources = {
+              'bar': 'raw/foo/evidence_bar.json',
+              'baz': 'raw/foo/evidence_baz.json'
+          }
+          target = 'foo/derived_bar_baz_{}.json'.format(system)
+          with derived_evidence(self.locker, sources, target) as evidences:
+              # None is returned if target evidence is not stale
+              if evidences:
+                  # Construct your derived evidence
+                  derived_data = self._do_whatever(
+                      evidences['bar'],
+                      evidences['baz']
+                  )
+                  # Set the content as a string
+                  # Upon exit it is written to the evidence locker
+                  evidences['derived'].set_content(json.dumps(derived_data))
+
+Usage example (source string provided)::
+
+  ...
+  from compliance.evidence import derived_evidence
+  ...
+  fetch_foo_bar_derived_evidence(self):
+      for system in systems:
+          source = 'raw/foo/evidence_bar.json'
+          target = 'foo/derived_bar_{}.json'.format(system)
+          with derived_evidence(self.locker, source, target) as evidences:
+              # None is returned if target evidence is not stale
+              if evidences:
+                  # Construct your derived evidence
+                  derived_data = self._do_whatever(evidences['source'])
+                  # Set the content as a string
+                  # Upon exit it is written to the evidence locker
+                  evidences['derived'].set_content(json.dumps(derived_data))
+
 Evidence Dependency Chaining
 ============================
 
@@ -355,22 +473,22 @@ we've provided some helpful decorators and context managers that validate
 ``ttl`` for you and will ``ERROR`` the check if evidence ``ttl`` has expired
 prior to executing the check's logic.
 
-* ``with_raw_evidences``, ``with_external_evidences`` decorators: Use these
-  decorators on your check method when you know the full path and name of
-  your raw or external evidence.  Each decorator takes as arguments, the
-  paths to your raw or external evidence as strings or as evidence
-  ``LazyLoader`` named tuples.  Evidence ``LazyLoader`` has ``path`` and
-  ``ev_class`` (evidence class) as attributes.  If the requested evidence pass
-  TTL validation the evidence is then passed along to the decorated method in
-  the form of method arguments.  Use an evidence ``LazyLoader`` when dealing
-  with sub-classed ``RawEvidence`` or ``ExternalEvidence``, and you want the
-  evidence provided to the decorated method to be cast as that sub-classed
+* ``with_raw_evidences``, ``with_derived_evidences``, ``with_tmp_evidences``,
+  and ``with_external_evidences`` decorators: Use these decorators on your
+  check method when you know the path and name of your raw, derived, tmp or
+  external evidence.  Each decorator takes as arguments, the paths to your  evidence as strings or as evidence ``LazyLoader`` named tuples.  Evidence
+  ``LazyLoader`` has ``path`` and ``ev_class`` (evidence class) as attributes.
+  If the requested evidence pass TTL validation the evidence is then passed
+  along to the decorated method in the form of method arguments.  Use an
+  evidence ``LazyLoader`` when dealing with sub-classed ``RawEvidence``,
+  ``DerivedEvidence``, ``TmpEvidence``, or ``ExternalEvidence``, and you want
+  the evidence provided to the decorated method to be cast as that sub-classed
   evidence otherwise use a string path and the evidence will be provided as
-  either ``RawEvidence`` or ``ExternalEvidence``.  A ``LazyLoader`` named tuple
-  can be constructed by executing the ``lazy_load`` class method of any evidence
+  the appropriate base evidence.  A ``LazyLoader`` named tuple can be
+  constructed by executing the ``lazy_load`` class method of any evidence
   class such as ``BarEvidence.lazy_load('foo/evidence_bar.json')``.
 
-``@with_*_evidences`` usage example::
+Usage example::
 
   ...
   from compliance.evidence import with_raw_evidences
@@ -413,7 +531,7 @@ prior to executing the check's logic.
   the ``lazy_load`` class method of any evidence class such as
   ``BarEvidence.lazy_load('foo/evidence_bar.json')``.
 
-``with evidences`` (list provided) usage example::
+Usage example (list provided)::
 
   ...
   from compliance.evidence import evidences
@@ -436,7 +554,7 @@ prior to executing the check's logic.
               self.add_warnings('bar vs. baz', warnings)
               self.add_successes('bar vs. baz', successes)
 
-``with evidences`` (dictionary provided) usage example::
+Usage example (dictionary provided)::
 
   ...
   from compliance.evidence import evidences
@@ -459,7 +577,7 @@ prior to executing the check's logic.
               self.add_warnings('bar vs. baz', warnings)
               self.add_successes('bar vs. baz', successes)
 
-``with evidences`` (string path provided) usage example::
+Usage example (string path provided)::
 
   ...
   from compliance.evidence import evidences
@@ -475,7 +593,7 @@ prior to executing the check's logic.
               self.add_warnings('bar stuff', warnings)
               self.add_successes('bar stuff', successes)
 
-``with evidences`` (``LazyLoader`` provided) usage example::
+Usage example (``LazyLoader`` provided)::
 
   ...
   from compliance.evidence import evidences

setup.cfg

@@ -25,7 +25,6 @@ install_requires =
     GitPython>=2.1.3
     jinja2>=2.10
     ibm_cloud_security_advisor>=2.0.0
-    deprecated>=1.2.9
     ilcli>=0.3.1
 
 [options.packages.find]