Merge pull request #1699 from jjotero/feature/extensible-vars

[feat] Add built-ins to manage test variables and parameters

Author: Vasileios Karakasis

docs/regression_test_api.rst

@@ -36,57 +36,168 @@ Pipeline Hooks
 .. autodecorator:: reframe.core.decorators.require_deps
 
 
-.. _directives:
 
-Directives
-----------
+Builtins
+--------
 
-Directives are functions that can be called directly in the body of a ReFrame regression test class. 
-These functions exert control over the test creation, and they allow adding and/or modifying certain attributes of the regression test.
-For example, a test can be parameterized using the :func:`parameter` directive as follows:
+.. versionadded:: 3.4.2
 
-.. code:: python
+ReFrame provides built-in functions that facilitate the creation of extensible tests (i.e. a test library).
+These *builtins* are intended to be used directly in the class body of the test, allowing the ReFrame internals to *pre-process* their input before the actual test creation takes place.
+This provides the ReFrame internals with further control over the user's input, making the process of writing regression tests less error-prone thanks to a better error checking.
+In essence, these builtins exert control over the test creation, and they allow adding and/or modifying certain attributes of the regression test.
+
+
+.. py:function:: reframe.core.pipeline.RegressionTest.parameter(values=None, inherit_params=False, filter_params=None)
+
+  Inserts or modifies a regression test parameter.
+  If a parameter with a matching name is already present in the parameter space of a parent class, the existing parameter values will be combined with those provided by this method following the inheritance behavior set by the arguments ``inherit_params`` and ``filter_params``.
+  Instead, if no parameter with a matching name exists in any of the parent parameter spaces, a new regression test parameter is created.
+  A regression test can be parametrized as follows:
+
+  .. code:: python
+
+    class Foo(rfm.RegressionTest):
+        variant = parameter(['A', 'B'])
 
-    class MyTest(rfm.RegressionTest):
-        parameter('variant', ['A', 'B'])
- 
         def __init__(self):
             if self.variant == 'A':
                 do_this()
             else:
                 do_other()
 
-One of the most powerful features about using directives is that they store their input information at the class level. 
-This means if one were to extend or specialize an existing regression test, the test attribute additions and modifications made through directives in the parent class will be automatically inherited by the child test.
-For instance, continuing with the example above, one could override the :func:`__init__` method in the :class:`MyTest` regression test as follows:
-
-.. code:: python
+  One of the most powerful features about these built-in functions is that they store their input information at the class level. 
+  This means if one were to extend or specialize an existing regression test, the test attribute additions and modifications made through built-in functions in the parent class will be automatically inherited by the child test.
+  For instance, continuing with the example above, one could override the :func:`__init__` method in the :class:`MyTest` regression test as follows:
 
-    class MyModifiedTest(MyTest):
+  .. code:: python
 
+    class Bar(Foo):
         def __init__(self):
             if self.variant == 'A':
                 override_this()
             else:
                 override_other()
 
+  Note that this built-in parameter function provides an alternative method to parameterize a test to :func:`reframe.core.decorators.parameterized_test`, and the use of both approaches in the same test is currently disallowed.
+  The two main advantages of the built-in :func:`parameter` over the decorated approach reside in the parameter inheritance across classes and the handling of large parameter sets.
+  As shown in the example above, the parameters declared with the built-in :func:`parameter` are automatically carried over into derived tests through class inheritance, whereas tests using the decorated approach would have to redefine the parameters on every test. 
+  Similarly, parameters declared through the built-in :func:`parameter` are regarded as fully independent from each other and ReFrame will automatically generate as many tests as available parameter combinations. This is a major advantage over the decorated approach, where one would have to manually expand the parameter combinations.
+  This is illustrated in the example below, consisting of a case with two parameters, each having two possible values.
+
+  .. code:: python
+
+    # Parameterized test with two parameters (p0 = ['a', 'b'] and p1 = ['x', 'y'])
+    @rfm.parameterized_test(['a','x'], ['a','y'], ['b','x'], ['b', 'y'])
+    class Foo(rfm.RegressionTest):
+      def __init__(self, p0, p1):
+        do_something(p0, p1)
+
+    # This is easier to write with the parameter built-in.
+    @rfm.simple_test
+    class Bar(rfm.RegressionTest):
+      p0 = parameter(['a', 'b'])
+      p1 = parameter(['x', 'y'])
+
+      def __init__(self):
+        do_something(self.p0, self.p1)
+
+
+  :param values: A list containing the parameter values.
+     If no values are passed when creating a new parameter, the parameter is considered as *declared* but not *defined* (i.e. an abstract parameter).
+     Instead, for an existing parameter, this depends on the parameter's inheritance behaviour and on whether any values where provided in any of the parent parameter spaces.
+  :param inherit_params: If :obj:`False`, no parameter values that may have been defined in any of the parent parameter spaces will be inherited.
+  :param filter_params: Function to filter/modify the inherited parameter values that may have been provided in any of the parent parameter spaces.
+     This function must accept a single argument, which will be passed as an iterable containing the inherited parameter values.
+     This only has an effect if used with ``inherit_params=True``.
+
+
+.. py:function:: reframe.core.pipeline.RegressionTest.variable(*types, value=None)
+
+  Inserts a new regression test variable.
+  Declaring a test variable through the :func:`variable` built-in allows for a more robust test implementation than if the variables were just defined as regular test attributes (e.g. ``self.a = 10``).
+  Using variables declared through the :func:`variable` built-in guarantees that these regression test variables will not be redeclared by any child class, while also ensuring that any values that may be assigned to such variables comply with its original declaration. 
+  In essence, by using test variables, the user removes any potential test errors that might be caused by accidentally overriding a class attribute. See the example below.
+
+
+  .. code:: python
+
+    class Foo(rfm.RegressionTest):
+        my_var = variable(int, value=8)
+        not_a_var = 4
+
+        def __init__(self):
+            print(self.my_var) # prints 8.
+            # self.my_var = 'override' # Error: my_var must be an int!
+            self.not_a_var = 'override' # However, this would work. Dangerous!
+            self.my_var = 10 # tests may also assign values the standard way
+
+  The argument ``value`` in the :func:`variable` built-in sets the default value for the variable.
+  As mentioned above, a variable may not be declared more than once, but its default value can be updated by simply assigning it a new value directly in the class body.
+
+  .. code:: python
+
+    class Bar(Foo):
+        my_var = 4
+        # my_var = 'override' # Error again!
+
+        def __init__(self):
+            print(self.my_var) # prints 4.
+
+  Here, the class :class:`Bar` inherits the variables from :class:`Foo` and can see that ``my_var`` has already been declared in the parent class. Therefore, the value of ``my_var`` is updated ensuring that the new value complies to the original variable declaration.
+
+  These examples above assumed that a default value can be provided to the variables in the bases tests, but that might not always be the case.
+  For example, when writing a test library, one might want to leave some variables undefined and force the user to set these when using the test.
+  As shown in the example below, imposing such requirement is as simple as not passing any ``value`` to the :func:`variable` built-in, which marks the given variable as *required*.
+
+  .. code:: python
+
+    # Test as written in the library
+    class EchoBaseTest(rfm.RunOnlyRegressionTest):
+      what = variable(str)
+
+      def __init__(self):
+          self.valid_systems = ['*']
+          self.valid_prog_environs = ['PrgEnv-gnu']
+          self.executable = f'echo {self.what}'
+          self.sanity_patterns = sn.assert_found(fr'{self.what}')
+
+
+    # Test as written by the user
+    @rfm.simple_test
+    class HelloTest(EchoBaseTest):
+      what = 'Hello'
+
+
+    # A parametrized test with type-checking
+    @rfm.simple_test
+    class FoodTest(EchoBaseTest):
+      param = parameter(['Bacon', 'Eggs'])
+
+      def __init__(self):
+        self.what = self.param
+        super().__init__()
+
+
+  Similarly to a variable with a value already assigned to it, the value of a required variable may be set either directly in the class body, on the :func:`__init__` method, or in any other hook before it is referenced.
+  Otherwise an error will be raised indicating that a required variable has not been set.
+  Conversely, a variable with a default value already assigned to it can be made required by assigning it the ``required`` keyword.
 
-.. py:function:: reframe.core.pipeline.RegressionTest.parameter(name, values=None, inherit_params=False, filter_params=None)
+  .. code:: python
 
-   Inserts or modifies a regression test parameter.
-   If a parameter with a matching name is already present in the parameter space of a parent class, the existing parameter values will be combined with those provided by this method following the inheritance behaviour set by the arguments ``inherit_params`` and ``filter_params``.
-   Instead, if no parameter with a matching name exists in any of the parent parameter spaces, a new regression test parameter is created.
+    class MyRequiredTest(HelloTest):
+      what = required
 
-   :param name: The parameter name.
-   :param values: A list containing the parameter values.
-       If no values are passed when creating a new parameter, the parameter is considered as *declared* but not *defined* (i.e. an abstract parameter).
-       Instead, for an existing parameter, this depends on the parameter's inheritance behaviour and on whether any values where provided in any of the parent parameter spaces.
-   :param inherit_params: If :obj:`False`, no parameter values that may have been defined in any of the parent parameter spaces will be inherited.
-   :param filter_params: Function to filter/modify the inherited parameter values that may have been provided in any of the parent parameter spaces.
-       This function must accept a single argument, which will be passed as an iterable containing the inherited parameter values.
-       This only has an effect if used with ``inherit_params=True``.
 
+  Running the above test will cause the :func:`__init__` method from :class:`EchoBaseTest` to throw an error indicating that the variable ``what`` has not been set.
 
+  :param types: the supported types for the variable.
+  :param value: the default value assigned to the variable. If no value is provided, the variable is set as ``required``.
+  :param field: the field validator to be used for this variable.
+      If no field argument is provided, it defaults to
+      :class:`reframe.core.fields.TypedField`.
+      Note that the field validator provided by this argument must derive from
+      :class:`reframe.core.fields.Field`.
 
 
 Environments and Systems

reframe/core/meta.py

@@ -9,29 +9,69 @@
 
 
 from reframe.core.exceptions import ReframeSyntaxError
+import reframe.core.namespaces as namespaces
 import reframe.core.parameters as parameters
+import reframe.core.variables as variables
 
 
 class RegressionTestMeta(type):
+
+    class MetaNamespace(namespaces.LocalNamespace):
+        '''Custom namespace to control the cls attribute assignment.'''
+        def __setitem__(self, key, value):
+            if isinstance(value, variables.VarDirective):
+                # Insert the attribute in the variable namespace
+                self['_rfm_local_var_space'][key] = value
+            elif isinstance(value, parameters.TestParam):
+                # Insert the attribute in the parameter namespace
+                self['_rfm_local_param_space'][key] = value
+            else:
+                super().__setitem__(key, value)
+
     @classmethod
-    def __prepare__(cls, name, bases, **kwargs):
+    def __prepare__(metacls, name, bases, **kwargs):
         namespace = super().__prepare__(name, bases, **kwargs)
 
         # Regression test parameter space defined at the class level
-        local_param_space = parameters.LocalParamSpace()
+        local_param_space = namespaces.LocalNamespace()
         namespace['_rfm_local_param_space'] = local_param_space
 
-        # Directive to add a regression test parameter directly in the
-        # class body as: `parameter('P0', 0,1,2,3)`.
-        namespace['parameter'] = local_param_space.add_param
+        # Directive to insert a regression test parameter directly in the
+        # class body as: `P0 = parameter([0,1,2,3])`.
+        namespace['parameter'] = parameters.TestParam
+
+        # Regression test var space defined at the class level
+        local_var_space = namespaces.LocalNamespace()
+        namespace['_rfm_local_var_space'] = local_var_space
 
-        return namespace
+        # Directives to add/modify a regression test variable
+        namespace['variable'] = variables.TestVar
+        namespace['required'] = variables.UndefineVar()
+        return metacls.MetaNamespace(namespace)
+
+    def __new__(metacls, name, bases, namespace, **kwargs):
+        return super().__new__(metacls, name, bases, dict(namespace), **kwargs)
 
     def __init__(cls, name, bases, namespace, **kwargs):
         super().__init__(name, bases, namespace, **kwargs)
 
-        # Build the regression test parameter space
-        cls._rfm_param_space = parameters.ParamSpace(cls)
+        # Create a set with the attribute names already in use.
+        cls._rfm_dir = set()
+        for base in bases:
+            if hasattr(base, '_rfm_dir'):
+                cls._rfm_dir.update(base._rfm_dir)
+
+        used_attribute_names = set(cls._rfm_dir)
+
+        # Build the var space and extend the target namespace
+        variables.VarSpace(cls, used_attribute_names)
+        used_attribute_names.update(cls._rfm_var_space.vars)
+
+        # Build the parameter space
+        parameters.ParamSpace(cls, used_attribute_names)
+
+        # Update used names set with the local __dict__
+        cls._rfm_dir.update(cls.__dict__)
 
         # Set up the hooks for the pipeline stages based on the _rfm_attach
         # attribute; all dependencies will be resolved first in the post-setup
@@ -100,6 +140,27 @@ def __call__(cls, *args, **kwargs):
         obj.__init__(*args, **kwargs)
         return obj
 
+    def __getattribute__(cls, name):
+        ''' Attribute lookup method for the MetaNamespace.
+
+        This metaclass implements a custom namespace, where built-in `variable`
+        and `parameter` types are stored in their own sub-namespaces (see
+        :class:`reframe.core.meta.RegressionTestMeta.MetaNamespace`).
+        This method will perform an attribute lookup on these sub-namespaces if
+        a call to the default `__getattribute__` method fails to retrieve the
+        requested class attribute.
+        '''
+        try:
+            return super().__getattribute__(name)
+        except AttributeError:
+            try:
+                return cls._rfm_local_var_space[name]
+            except KeyError:
+                try:
+                    return cls._rfm_local_param_space[name]
+                except KeyError:
+                    return super().__getattr__(name)
+
     @property
     def param_space(cls):
         # Make the parameter space available as read-only