Merge branch 'master' into ci/base_dockerfiles

Author: Theofilos Manitaras

README.md

@@ -13,7 +13,7 @@
 [![Slack](https://reframe-slack.herokuapp.com/badge.svg)](https://reframe-slack.herokuapp.com/)<br/>
 [![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)
 [![DOI](https://zenodo.org/badge/89384186.svg)](https://zenodo.org/badge/latestdoi/89384186)<br/>
-![Twitter Follow](https://img.shields.io/twitter/follow/ReFrameHPC?style=social)
+[![Twitter Follow](https://img.shields.io/twitter/follow/ReFrameHPC?style=social)](https://twitter.com/ReFrameHPC)
 
 # ReFrame in a Nutshell
 

cscs-checks/system/slurm/slurm.py

@@ -257,3 +257,36 @@ def reference_meminfo(self):
             'pilatus:mc': 250,
         }
         return reference_meminfo[self.current_partition.fullname]
+
+
+@rfm.simple_test
+class slurm_response_check(rfm.RunOnlyRegressionTest):
+    command = parameter(['squeue', 'sacct'])
+    descr = 'Slurm command test'
+    valid_systems = ['daint:login', 'dom:login']
+    valid_prog_environs = ['builtin']
+    num_tasks = 1
+    num_tasks_per_node = 1
+    reference = {
+        'squeue': {
+            'real_time': (0.02, None, 0.1, 's')
+        },
+        'sacct': {
+            'real_time': (0.1, None, 0.1, 's')
+        }
+    }
+    executable = 'time -p'
+    tags = {'diagnostic', 'health'}
+    maintainers = ['CB', 'VH']
+
+    @rfm.run_before('run')
+    def set_exec_opts(self):
+        self.executable_opts = [self.command]
+
+    @rfm.run_before('sanity')
+    def set_sanity(self):
+        self.sanity_patterns = sn.assert_eq(self.job.exitcode, 0)
+        self.perf_patterns = {
+            'real_time': sn.extractsingle(r'real (?P<real_time>\S+)',
+                                          self.stderr, 'real_time', float)
+        }

docs/_static/img/favicon.png

@@ -29,6 +29,23 @@ Regression Test Class Decorators
 Pipeline Hooks
 --------------
 
+.. versionadded:: 2.20
+
+
+Pipeline hooks is an easy way to perform operations while the test traverses the execution pipeline.
+You can attach arbitrary functions to run before or after any pipeline stage, which are called *pipeline hooks*.
+Multiple hooks can be attached before or after the same pipeline stage, in which case the order of execution will match the order in which the functions are defined in the class body of the test.
+A single hook can also be applied to multiple stages and it will be executed multiple times.
+All pipeline hooks of a test class are inherited by its subclasses.
+Subclasses may override a pipeline hook of their parents by redefining the hook function and re-attaching it at the same pipeline stage.
+There are seven pipeline stages where you can attach test methods: ``init``, ``setup``, ``compile``, ``run``, ``sanity``, ``performance`` and ``cleanup``.
+The ``init`` stage is not a real pipeline stage, but it refers to the test initialization.
+
+Hooks attached to any stage will run exactly before or after this stage executes.
+So although a "post-init" and a "pre-setup" hook will both run *after* a test has been initialized and *before* the test goes through the first pipeline stage, they will execute in different times:
+the post-init hook will execute *right after* the test is initialized.
+The framework will then continue with other activities and it will execute the pre-setup hook *just before* it schedules the test for executing its setup stage.
+
 .. autodecorator:: reframe.core.decorators.run_after(stage)
 
 .. autodecorator:: reframe.core.decorators.run_before(stage)

docs/regression_test_api.rst

@@ -790,3 +790,63 @@ Therefore, the ``release.txt`` file can now be used in the subsequent sanity che
 
 For a complete list of the available attributes of a specific container platform, please have a look at the :ref:`container-platforms` section of the :doc:`regression_test_api` guide.
 On how to configure ReFrame for running containerized tests, please have a look at the :ref:`container-platform-configuration` section of the :doc:`config_reference`.
+
+
+Writing reusable tests
+----------------------
+
+.. versionadded:: 3.5.0
+
+So far, all the examples shown above were tight to a particular system or configuration, which makes reusing these tests in other systems not straightforward.
+However, the introduction of the :py:func:`~reframe.core.pipeline.RegressionTest.parameter` and :py:func:`~reframe.core.pipeline.RegressionTest.variable` ReFrame built-ins solves this problem, eliminating the need to specify any of the test variables in the :func:`__init__` method.
+Hence, these parameters and variables can be treated as simple class attributes, which allows us to leverage Python's class inheritance and write more modular tests.
+For simplicity, we illustrate this concept with the above :class:`ContainerTest` example, where the goal here is to re-write this test as a library that users can simply import from and derive their tests without having to rewrite the bulk of the test.
+Also, for illustrative purposes, we parameterize this library test on a few different image tags (the above example just used ``ubuntu:18.04``) and throw the container commands into a separate bash script just to create some source files.
+Thus, removing all the system and configuration specific variables, and moving as many assignments as possible into the class body, the system agnostic library test looks as follows:
+
+.. code-block:: console
+
+   cat tutorials/advanced/library/lib/__init__.py
+
+
+.. literalinclude:: ../tutorials/advanced/library/lib/__init__.py
+   :lines: 6-
+   :emphasize-lines: 8-17
+
+Note that the class :class:`ContainerBase` is not decorated since it does not specify the required variables ``valid_systems`` and ``valid_prog_environs``, and it declares the ``platform`` parameter without any defined values assigned.
+Hence, the user can simply derive from this test and specialize it to use the desired container platforms.
+Since the parameters are defined directly in the class body, the user is also free to override or extend any of the other parameters in a derived test.
+In this example, we have parametrized the base test to run with the ``ubuntu:18.04`` and ``ubuntu:20.04`` images, but these values from ``dist`` (and also the ``dist_name`` variable) could be modified by the derived class if needed.
+
+On the other hand, the rest of the test depends on the values from the test parameters, and a parameter is only assigned a specific value after the class has been instantiated.
+Thus, the rest of the test is expressed as hooks, without the need to write anything in the :func:`__init__` method.
+In fact, writing the test in this way permits having hooks that depend on undefined variables or parameters.
+This is the case with the :func:`set_container_platform` hook, which depends on the undefined parameter ``platform``.
+Hence, the derived test **must** define all the required parameters and variables; otherwise ReFrame will notice that the test is not well defined and will raise an error accordingly.
+
+Before moving onwards to the derived test, note that the :class:`ContainerBase` class takes the additional argument ``pin_prefix=True``, which locks the prefix of all derived tests to this base test.
+This will allow the retrieval of the sources located in the library by any derived test, regardless of what their containing directory is.
+
+.. code-block:: console
+
+   cat tutorials/advanced/library/lib/src/get_os_release.sh
+
+
+.. literalinclude:: ../tutorials/advanced/library/lib/src/get_os_release.sh
+   :lines: 1-
+
+Now from the user's perspective, the only thing to do is to import the above base test and specify the required variables and parameters.
+For consistency with the above example, we set the ``platform`` parameter to use Sarus and Singularity, and we configure the test to run on Piz Daint with the built-in programming environment.
+Hence, the above :class:`ContainerTest` is now reduced to the following:
+
+.. code-block:: console
+
+   cat tutorials/advanced/library/usr/container_test.py
+
+
+.. literalinclude:: ../tutorials/advanced/library/usr/container_test.py
+   :lines: 6-
+
+In a similar fashion, any other user could reuse the above :class:`ContainerBase` class and write the test for their own system with a few lines of code.
+
+*Happy test sharing!*

docs/tutorial_advanced.rst

@@ -417,7 +417,7 @@ We extend our C++ "Hello, World!" example to print the greetings from multiple t
    :language: cpp
    :lines: 6-
 
-This program takes as argument the number of threads it will create and it uses ``std::thread``, which is C++11 addition, meaning that we will need to pass ``-std=c++11`` to our compilers.
+This program takes as argument the number of threads it will create and it uses ``std::thread``, which is a C++11 addition, meaning that we will need to pass ``-std=c++11`` to our compilers.
 Here is the corresponding ReFrame test, where the new concepts introduced are highlighted:
 
 .. code-block:: console
@@ -429,12 +429,30 @@ Here is the corresponding ReFrame test, where the new concepts introduced are hi
    :lines: 6-
    :emphasize-lines: 11-13
 
+
+In order to compile applications using ``std::thread`` with GCC and Clang, the ``-pthread`` option has to be passed to the compiler.
+Since the above option might not be valid for other compilers, we use pipeline hooks to differentiate based on the programming environment as follows:
+
+.. code-block:: python
+
+   @rfm.run_before('compile')
+   def set_threading_flags(self):
+       environ = self.current_environ.name
+       if environ in {'clang', 'gnu'}:
+           self.build_system.cxxflags += ['-pthread']
+
+
+.. note::
+
+   The pipeline hooks, as well as the regression test pipeline itself, are covered in more detail later on in the tutorial.
+
+
 ReFrame delegates the compilation of a test to a *build system*, which is an abstraction of the steps needed to compile the test.
 Build systems take also care of interactions with the programming environment if necessary.
 Compilation flags are a property of the build system.
 If not explicitly specified, ReFrame will try to pick the correct build system (e.g., CMake, Autotools etc.) by inspecting the test resources, but in cases as the one presented here where we need to set the compilation flags, we need to specify a build system explicitly.
-In this example, we instruct ReFrame to compile a single source file using the ``-std=c++11 -Wall`` compilation flags.
-Finally, we set the arguments to be passed to the generated executable in :attr:`~reframe.core.pipeline.RegressionTest.executable_opts`.
+In this example, we instruct ReFrame to compile a single source file using the ``-std=c++11 -pthread -Wall`` compilation flags.
+Finally, we set the arguments to be passed to the generated executable in :attr:`executable_opts <reframe.core.pipeline.RegressionTest.executable_opts>`.
 
 
 .. code-block:: console
@@ -1075,7 +1093,7 @@ Let's see and comment the changes:
 First of all, we need to add the new programming environments in the list of the supported ones.
 Now there is the problem that each compiler has its own flags for enabling OpenMP, so we need to differentiate the behavior of the test based on the programming environment.
 For this reason, we define the flags for each compiler in a separate dictionary (``self.flags``) and we set them in the :func:`setflags` pipeline hook.
-Let's explain what is this all about.
+We have first seen the pipeline hooks in the multithreaded "Hello, World!" example and now we explain them in more detail.
 When ReFrame loads a test file, it instantiates all the tests it finds in it.
 Based on the system ReFrame runs on and the supported environments of the tests, it will generate different test cases for each system partition and environment combination and it will finally send the test cases for execution.
 During its execution, a test case goes through the *regression test pipeline*, which is a series of well defined phases.

docs/tutorial_basics.rst

@@ -6,7 +6,7 @@
 import os
 import sys
 
-VERSION = '3.6.0-dev.1'
+VERSION = '3.6.0-dev.2'
 INSTALL_PREFIX = os.path.normpath(
     os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
 )

reframe/__init__.py

@@ -147,7 +147,11 @@ def post_build(self, buildjob):
         Build systems may use this information to do some post processing and
         provide additional, build system-specific, functionality to the users.
 
+        This function will always be executed from the test's stage directory.
+
         .. versionadded:: 3.5.0
+        .. versionchanged:: 3.5.2
+           The function is executed from the stage directory.
 
         :meta private:
 
@@ -741,7 +745,6 @@ def __init__(self):
         self.package_opts = {}
         self.prefix = 'easybuild'
         self._eb_modules = []
-        self._prefix_save = None
 
     def emit_build_commands(self, environ):
         if not self.easyconfigs:
@@ -755,7 +758,6 @@ def emit_build_commands(self, environ):
 
         prefix = os.path.join(os.getcwd(), self.prefix)
         options = ' '.join(self.options)
-        self._prefix_save = prefix
         return [f'export EASYBUILD_BUILDPATH={prefix}/build',
                 f'export EASYBUILD_INSTALLPATH={prefix}',
                 f'export EASYBUILD_PREFIX={prefix}',
@@ -765,7 +767,8 @@ def emit_build_commands(self, environ):
     def post_build(self, buildjob):
         # Store the modules generated by EasyBuild
 
-        modulesdir = os.path.join(self._prefix_save, 'modules', 'all')
+        modulesdir = os.path.join(os.getcwd(), self.prefix,
+                                  'modules', 'all')
         with open(buildjob.stdout) as fp:
             out = fp.read()
 

reframe/core/buildsystems.py

@@ -85,8 +85,8 @@ def _validate_test(cls):
                                  'subclass of RegressionTest')
 
     if (cls.is_abstract()):
-        raise ValueError(f'decorated test ({cls.__qualname__!r}) is an'
-                         f' abstract test')
+        raise ValueError(f'decorated test ({cls.__qualname__!r}) has one or '
+                         f'more undefined parameters')
 
 
 def simple_test(cls):
@@ -214,6 +214,13 @@ def _fn(*args, **kwargs):
     return deco
 
 
+# Valid pipeline stages that users can specify in the `run_before()` and
+# `run_after()` decorators
+_USER_PIPELINE_STAGES = (
+    'init', 'setup', 'compile', 'run', 'sanity', 'performance', 'cleanup'
+)
+
+
 def run_before(stage):
     '''Decorator for attaching a test method to a pipeline stage.
 
@@ -226,19 +233,57 @@ def run_before(stage):
     The ``stage`` argument can be any of ``'setup'``, ``'compile'``,
     ``'run'``, ``'sanity'``, ``'performance'`` or ``'cleanup'``.
 
-    .. versionadded:: 2.20
     '''
+    if stage not in _USER_PIPELINE_STAGES:
+        raise ValueError(f'invalid pipeline stage specified: {stage!r}')
+
+    if stage == 'init':
+        raise ValueError('pre-init hooks are not allowed')
+
     return _runx('pre_' + stage)
 
 
 def run_after(stage):
     '''Decorator for attaching a test method to a pipeline stage.
 
-    This is completely analogous to the
-    :py:attr:`reframe.core.decorators.run_before`.
+    This is analogous to the :py:attr:`~reframe.core.decorators.run_before`,
+    except that ``'init'`` can also be used as the ``stage`` argument. In this
+    case, the hook will execute right after the test is initialized (i.e.
+    after the :func:`__init__` method is called), before entering the test's
+    pipeline. In essence, a post-init hook is equivalent to defining
+    additional :func:`__init__` functions in the test. All the other
+    properties of pipeline hooks apply equally here. The following code
+
+    .. code-block:: python
+
+       @rfm.run_after('init')
+       def foo(self):
+           self.x = 1
+
+
+    is equivalent to
+
+    .. code-block:: python
+
+       def __init__(self):
+           self.x = 1
+
+    .. versionchanged:: 3.5.2
+       Add the ability to define post-init hooks in tests.
 
-    .. versionadded:: 2.20
     '''
+
+    if stage not in _USER_PIPELINE_STAGES:
+        raise ValueError(f'invalid pipeline stage specified: {stage!r}')
+
+    # Map user stage names to the actual pipeline functions if needed
+    if stage == 'init':
+        stage = '__init__'
+    elif stage == 'compile':
+        stage = 'compile_wait'
+    elif stage == 'run':
+        stage = 'run_wait'
+
     return _runx('post_' + stage)