Merge branch 'master' into doc/fixtures-tutorial

Author: Vasileios Karakasis

README.md

@@ -14,17 +14,15 @@
 
 # ReFrame in a Nutshell
 
-ReFrame is a framework for writing regression tests for HPC systems.
-The goal of this framework is to abstract away the complexity of the interactions with the system, separating the logic of a regression test from the low-level details, which pertain to the system configuration and setup.
-This allows users to write easily portable regression tests, focusing only on the functionality.
+ReFrame is a powerful framework for writing system regression tests and benchmarks, specifically targeted to HPC systems.
+The goal of the framework is to abstract away the complexity of the interactions with the system, separating the logic of a test from the low-level details, which pertain to the system configuration and setup.
+This allows users to write portable tests in a declarative way that describes only the test's functionality.
 
-Regression tests in ReFrame are simple Python classes that specify the basic parameters of the test.
-The framework will load the test and will send it down a well-defined pipeline that will take care of its execution.
+Tests in ReFrame are simple Python classes that specify the basic variables and parameters of the test.
+ReFrame offers an intuitive and very powerful syntax that allows users to create test libraries, test factories, as well as complete test workflows using other tests as fixtures.
+ReFrame will load the tests and send them down a well-defined pipeline that will execute them in parallel.
 The stages of this pipeline take care of all the system interaction details, such as programming environment switching, compilation, job submission, job status query, sanity checking and performance assessment.
 
-Writing system regression tests in a high-level modern programming language, like Python, poses a great advantage in organizing and maintaining the tests.
-Users can create their own test hierarchies, create test factories for generating multiple tests at the same time and also customize them in a simple and expressive way.
-
 Please visit the project's documentation [page](https://reframe-hpc.readthedocs.io/) for all the details!
 
 

docs/index.rst

@@ -2,22 +2,19 @@
 Welcome to ReFrame
 ==================
 
-ReFrame is a high-level framework for writing regression tests for HPC systems.
-The goal of the framework is to abstract away the complexity of the interactions with the system, separating the logic of a regression test from the low-level details, which pertain to the system configuration and setup.
-This allows users to write easily portable regression tests, focusing only on the functionality.
+ReFrame is a powerful framework for writing system regression tests and benchmarks, specifically targeted to HPC systems.
+The goal of the framework is to abstract away the complexity of the interactions with the system, separating the logic of a test from the low-level details, which pertain to the system configuration and setup.
+This allows users to write portable tests in a declarative way that describes only the test's functionality.
 
-Regression tests in ReFrame are simple Python classes that specify the basic parameters of the test.
-The framework will load the test and will send it down a well-defined pipeline that will take care of its execution.
+Tests in ReFrame are simple Python classes that specify the basic variables and parameters of the test.
+ReFrame offers an intuitive and very powerful syntax that allows users to create test libraries, test factories, as well as complete test workflows using other tests as fixtures.
+ReFrame will load the tests and send them down a well-defined pipeline that will execute them in parallel.
 The stages of this pipeline take care of all the system interaction details, such as programming environment switching, compilation, job submission, job status query, sanity checking and performance assessment.
 
 ReFrame also offers a high-level and flexible abstraction for writing sanity and performance checks for your regression tests, without having to care about the details of parsing output files, searching for patterns and testing against reference values for different systems.
 
-Writing system regression tests in a high-level modern programming language, like Python, poses a great advantage in organizing and maintaining the tests.
-Users can create their own test hierarchies or test factories for generating multiple tests at the same time and they can also customize them in a simple and expressive way.
-
 Finally, ReFrame offers a powerful and efficient runtime for running and managing the execution of tests, as well as integration with common logging facilities, where ReFrame can send live data from currently running performance tests.
 
-
 Use Cases
 =========
 

reframe/core/fixtures.py

@@ -527,16 +527,10 @@ class FixtureSpace(namespaces.Namespace):
     into the target instance under the ``_rfm_fixture_registry`` attribute.
     '''
 
-    @property
-    def local_namespace_name(self):
-        return '_rfm_local_fixture_space'
-
-    @property
-    def namespace_name(self):
-        return '_rfm_fixture_space'
-
-    def __init__(self, target_cls=None, target_namespace=None):
-        super().__init__(target_cls, target_namespace)
+    def __init__(self, target_cls=None, illegal_names=None):
+        super().__init__(target_cls, illegal_names,
+                         ns_name='_rfm_fixture_space',
+                         ns_local_name='_rfm_local_fixture_space')
 
         # Store all fixture variant combinations to allow random access.
         self.__variant_combinations = tuple(
@@ -563,7 +557,7 @@ def join(self, other, cls):
 
     def extend(self, cls):
         '''Extend the inherited fixture space with the local fixture space.'''
-        local_fixture_space = getattr(cls, self.local_namespace_name)
+        local_fixture_space = getattr(cls, self.local_namespace_name, False)
         while local_fixture_space:
             name, fixture = local_fixture_space.popitem()
             self.fixtures[name] = fixture

reframe/core/meta.py

@@ -393,18 +393,18 @@ def __init__(cls, name, bases, namespace, **kwargs):
         for base in (b for b in bases if hasattr(b, '_rfm_dir')):
             cls._rfm_dir.update(base._rfm_dir)
 
-        used_attribute_names = set(cls._rfm_dir)
-
-        # Build the var space and extend the target namespace
-        variables.VarSpace(cls, used_attribute_names)
-        used_attribute_names.update(cls._rfm_var_space.vars)
-
-        # Build the parameter space
-        parameters.ParamSpace(cls, used_attribute_names)
-        used_attribute_names.update(cls._rfm_param_space.params)
+        used_attribute_names = set(cls._rfm_dir).union(
+            {h.__name__ for h in cls._rfm_hook_registry}
+        )
 
-        # Build the fixture space
-        fixtures.FixtureSpace(cls, used_attribute_names)
+        # Build the different global class namespaces
+        namespace_types = (variables.VarSpace,
+                           parameters.ParamSpace,
+                           fixtures.FixtureSpace)
+        for ns_type in namespace_types:
+            ns = ns_type(cls, used_attribute_names)
+            setattr(cls, ns.namespace_name, ns)
+            used_attribute_names.update(ns.data())
 
         # Update used names set with the local __dict__
         cls._rfm_dir.update(cls.__dict__)

reframe/core/namespaces.py

@@ -68,22 +68,23 @@ def _raise_namespace_clash(self, name):
     def clear(self):
         self._namespace = {}
 
+    def data(self):
+        '''Give access to the underlying namespace'''
+        return self._namespace
+
 
 class Namespace(LocalNamespace, metaclass=abc.ABCMeta):
     '''Namespace of a regression test.
 
-    The final namespace may be built by inheriting namespaces from
-    the base classes, and extended with the information stored in the local
-    namespace of the target class. In this context, the target class is
-    simply the regression test class where the namespace is to be built.
-
-    To allow for this inheritance and extension of the namespace, this
-    class must define the names under which the local and final namespaces
-    are inserted in the target classes.
+    The final namespace may be built by inheriting namespaces from the base
+    classes, and extending this one with the information stored in the local
+    namespace of the target class. In this context, the target class is simply
+    the regression test class where the namespace is to be built.
 
-    If a target class is provided, the constructor will attach the Namespace
-    instance into the target class with the class attribute name as defined
-    in ``namespace_name``.
+    If a target class is provided, the constructor will build a Namespace
+    instance by inheriting the namespaces found in the base classes, and
+    extending this with the information from the local namespace of the
+    target class.
 
     Eventually, the items from a Namespace are injected as attributes of
     the target class instance by the :func:`inject` method, which must be
@@ -95,32 +96,15 @@ class may use more that one Namespace, which raises the need for name
     target class. Then, after the Namespace is built, if ``illegal_names`` is
     provided, a sanity check is performed, ensuring that no name clashing
     will occur during the target class instantiation process.
-    '''
 
-    @property
-    @abc.abstractmethod
-    def local_namespace_name(self):
-        '''Name of the local namespace in the target class.
-
-        Name under which the local namespace is stored in the
-        :class:`reframe.core.pipeline.RegressionTest` class.
-        '''
-
-    @property
-    @abc.abstractmethod
-    def namespace_name(self):
-        '''Name of the namespace in the target class.
-
-        Name under which the namespace is stored in the
-        :class:`reframe.core.pipeline.RegressionTest` class.
-        '''
+    '''
 
-    def __init__(self, target_cls=None, illegal_names=None):
+    def __init__(self, target_cls=None, illegal_names=None,
+                 *, ns_name, ns_local_name):
         super().__init__()
+        self._ns_name = ns_name
+        self._ns_local_name = ns_local_name
         if target_cls:
-            # Assert the Namespace can be built for the target_cls
-            self.assert_target_cls(target_cls)
-
             # Inherit Namespaces from the base clases
             self.inherit(target_cls)
 
@@ -130,23 +114,21 @@ def __init__(self, target_cls=None, illegal_names=None):
             # Sanity checkings on the resulting Namespace
             self.sanity(target_cls, illegal_names)
 
-            # Attach the Namespace to the target class
-            setattr(target_cls, self.namespace_name, self)
-
-    def assert_target_cls(self, cls):
-        '''Assert the target class has a valid local namespace.'''
+    @property
+    def namespace_name(self):
+        return self._ns_name
 
-        assert hasattr(cls, self.local_namespace_name)
-        assert isinstance(getattr(cls, self.local_namespace_name),
-                          LocalNamespace)
+    @property
+    def local_namespace_name(self):
+        return self._ns_local_name
 
     def inherit(self, cls):
         '''Inherit the Namespaces from the bases.'''
 
-        for base in filter(lambda x: hasattr(x, self.namespace_name),
-                           cls.__bases__):
-            assert isinstance(getattr(base, self.namespace_name), type(self))
-            self.join(getattr(base, self.namespace_name), cls)
+        for base in cls.__bases__:
+            other = getattr(base, self.namespace_name, None)
+            if isinstance(other, type(self)):
+                self.join(other, cls)
 
     @abc.abstractmethod
     def join(self, other, cls):
@@ -156,7 +138,7 @@ def join(self, other, cls):
     def extend(self, cls):
         '''Extend the namespace with the local namespace.'''
 
-    def sanity(self, cls, illegal_names=None):
+    def sanity(self, cls, illegal_names):
         '''Sanity checks post-creation of the namespace.
 
         By default, we make illegal to have any item in the namespace

reframe/core/parameters.py

@@ -61,43 +61,24 @@ def filter_params(x):
 
 
 class ParamSpace(namespaces.Namespace):
-    ''' Regression test parameter space
-
-    Host class for the parameter space of a regresion test. The parameter
-    space is stored as a dictionary (self.params), where the keys are the
-    parameter names and the values are tuples with all the available values
-    for each parameter. The __init__ method in this class takes an optional
-    argument (target_class), which is the regression test class where the
-    parameter space is to e inserted as the ``_rfm_param_space`` class
-    attribute. If no target class is provided, the parameter space is
-    initialized as empty. After the parameter space is set, a parameter space
-    iterator is created under self.__unique_iter, which acts as an internal
-    control variable that tracks the usage of this parameter space. This
-    iterator walks through all possible parameter combinations and cannot be
-    restored after reaching exhaustion. The length of this iterator matches
-    the value returned by the member function __len__.
-
-    :param target_cls: the class where the full parameter space is to be built.
-    :param target_namespace: a reference namespace to ensure that no name
-        clashes occur (see :class:`reframe.core.namespaces.Namespace`).
-
-    .. note::
-        The __init__ method is aware of the implementation details of the
-        regression test metaclass. This is required to retrieve the parameter
-        spaces from the base classes, and also the local parameter space from
-        the target class.
+    '''Regression test parameter space
+
+    The parameter space is stored as a dictionary (self.params), where the
+    keys are the parameter names and the values are tuples with all the
+    available values for each parameter. The __init__ method in this class
+    takes the optional argument ``target_cls``, which is the regression test
+    class that the parameter space is being built for. If no target class is
+    provided, the parameter space is initialized as empty.
+
+    All the parameter combinations are stored under ``__param_combinations``.
+    This enables random-access to any of the available parameter combinations
+    through the ``__getitem__`` method.
     '''
 
-    @property
-    def local_namespace_name(self):
-        return '_rfm_local_param_space'
-
-    @property
-    def namespace_name(self):
-        return '_rfm_param_space'
-
-    def __init__(self, target_cls=None, target_namespace=None):
-        super().__init__(target_cls, target_namespace)
+    def __init__(self, target_cls=None, illegal_names=None):
+        super().__init__(target_cls, illegal_names,
+                         ns_name='_rfm_param_space',
+                         ns_local_name='_rfm_local_param_space')
 
         # Store all param combinations to allow random access.
         self.__param_combinations = tuple(
@@ -142,7 +123,7 @@ def join(self, other, cls):
     def extend(self, cls):
         '''Extend the parameter space with the local parameter space.'''
 
-        local_param_space = getattr(cls, self.local_namespace_name)
+        local_param_space = getattr(cls, self.local_namespace_name, dict())
         for name, p in local_param_space.items():
             try:
                 filt_vals = p.filter_params(self.params.get(name, ()))

reframe/core/variables.py

@@ -425,30 +425,21 @@ def __ceil__(self):
 class VarSpace(namespaces.Namespace):
     '''Variable space of a regression test.
 
-    Store the variables of a regression test. This variable space is stored
-    in the regression test class under the class attribute ``_rfm_var_space``.
     A target class can be provided to the
     :func:`__init__` method, which is the regression test where the
     VarSpace is to be built. During this call to
     :func:`__init__`, the VarSpace inherits all the VarSpace from the base
     classes of the target class. After this, the VarSpace is extended with
-    the information from the local variable space, which is stored under the
-    target class' attribute ``_rfm_local_var_space``. If no target class is
+    the information from the local variable space. If no target class is
     provided, the VarSpace is simply initialized as empty.
     '''
 
-    @property
-    def local_namespace_name(self):
-        return '_rfm_local_var_space'
-
-    @property
-    def namespace_name(self):
-        return '_rfm_var_space'
-
     def __init__(self, target_cls=None, illegal_names=None):
         # Set to register the variables already injected in the class
         self._injected_vars = set()
-        super().__init__(target_cls, illegal_names)
+        super().__init__(target_cls, illegal_names,
+                         ns_name='_rfm_var_space',
+                         ns_local_name='_rfm_local_var_space')
 
     def join(self, other, cls):
         '''Join an existing VarSpace into the current one.
@@ -482,7 +473,7 @@ def extend(self, cls):
         of these actions on the same var for the same local var space
         is disallowed.
         '''
-        local_varspace = getattr(cls, self.local_namespace_name)
+        local_varspace = getattr(cls, self.local_namespace_name, False)
         while local_varspace:
             key, var = local_varspace.popitem()
             if isinstance(var, TestVar):
@@ -513,7 +504,7 @@ def extend(self, cls):
         for key in _assigned_vars:
             delattr(cls, key)
 
-    def sanity(self, cls, illegal_names=None):
+    def sanity(self, cls, illegal_names):
         '''Sanity checks post-creation of the var namespace.
 
         By default, we make illegal to have any item in the namespace

setup.cfg

@@ -2,7 +2,7 @@
 name = ReFrame-HPC
 version = attr: reframe.VERSION
 author = CSCS Swiss National Supercomputing Center
-description = ReFrame is a framework for writing regression tests for HPC systems
+description = ReFrame is a powerful framework for writing system regression tests and benchmarks, specifically targeted to HPC systems
 url = https://github.com/eth-cscs/reframe
 license = BSD 3-Clause
 long_description = file: README.md