You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/regression_test_api.rst
+17Lines changed: 17 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -29,6 +29,23 @@ Regression Test Class Decorators
29
29
Pipeline Hooks
30
30
--------------
31
31
32
+
.. versionadded:: 2.20
33
+
34
+
35
+
Pipeline hooks is an easy way to perform operations while the test traverses the execution pipeline.
36
+
You can attach arbitrary functions to run before or after any pipeline stage, which are called *pipeline hooks*.
37
+
Multiple hooks can be attached before or after the same pipeline stage, in which case the order of execution will match the order in which the functions are defined in the class body of the test.
38
+
A single hook can also be applied to multiple stages and it will be executed multiple times.
39
+
All pipeline hooks of a test class are inherited by its subclasses.
40
+
Subclasses may override a pipeline hook of their parents by redefining the hook function and re-attaching it at the same pipeline stage.
41
+
There are seven pipeline stages where you can attach test methods: ``init``, ``setup``, ``compile``, ``run``, ``sanity``, ``performance`` and ``cleanup``.
42
+
The ``init`` stage is not a real pipeline stage, but it refers to the test initialization.
43
+
44
+
Hooks attached to any stage will run exactly before or after this stage executes.
45
+
So although a "post-init" and a "pre-setup" hook will both run *after* a test has been initialized and *before* the test goes through the first pipeline stage, they will execute in different times:
46
+
the post-init hook will execute *right after* the test is initialized.
47
+
The framework will then continue with other activities and it will execute the pre-setup hook *just before* it schedules the test for executing its setup stage.
Copy file name to clipboardExpand all lines: docs/tutorial_advanced.rst
+60Lines changed: 60 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -790,3 +790,63 @@ Therefore, the ``release.txt`` file can now be used in the subsequent sanity che
790
790
791
791
For a complete list of the available attributes of a specific container platform, please have a look at the :ref:`container-platforms` section of the :doc:`regression_test_api` guide.
792
792
On how to configure ReFrame for running containerized tests, please have a look at the :ref:`container-platform-configuration` section of the :doc:`config_reference`.
793
+
794
+
795
+
Writing reusable tests
796
+
----------------------
797
+
798
+
.. versionadded:: 3.5.0
799
+
800
+
So far, all the examples shown above were tight to a particular system or configuration, which makes reusing these tests in other systems not straightforward.
801
+
However, the introduction of the :py:func:`~reframe.core.pipeline.RegressionTest.parameter` and :py:func:`~reframe.core.pipeline.RegressionTest.variable` ReFrame built-ins solves this problem, eliminating the need to specify any of the test variables in the :func:`__init__` method.
802
+
Hence, these parameters and variables can be treated as simple class attributes, which allows us to leverage Python's class inheritance and write more modular tests.
803
+
For simplicity, we illustrate this concept with the above :class:`ContainerTest` example, where the goal here is to re-write this test as a library that users can simply import from and derive their tests without having to rewrite the bulk of the test.
804
+
Also, for illustrative purposes, we parameterize this library test on a few different image tags (the above example just used ``ubuntu:18.04``) and throw the container commands into a separate bash script just to create some source files.
805
+
Thus, removing all the system and configuration specific variables, and moving as many assignments as possible into the class body, the system agnostic library test looks as follows:
Note that the class :class:`ContainerBase` is not decorated since it does not specify the required variables ``valid_systems`` and ``valid_prog_environs``, and it declares the ``platform`` parameter without any defined values assigned.
817
+
Hence, the user can simply derive from this test and specialize it to use the desired container platforms.
818
+
Since the parameters are defined directly in the class body, the user is also free to override or extend any of the other parameters in a derived test.
819
+
In this example, we have parametrized the base test to run with the ``ubuntu:18.04`` and ``ubuntu:20.04`` images, but these values from ``dist`` (and also the ``dist_name`` variable) could be modified by the derived class if needed.
820
+
821
+
On the other hand, the rest of the test depends on the values from the test parameters, and a parameter is only assigned a specific value after the class has been instantiated.
822
+
Thus, the rest of the test is expressed as hooks, without the need to write anything in the :func:`__init__` method.
823
+
In fact, writing the test in this way permits having hooks that depend on undefined variables or parameters.
824
+
This is the case with the :func:`set_container_platform` hook, which depends on the undefined parameter ``platform``.
825
+
Hence, the derived test **must** define all the required parameters and variables; otherwise ReFrame will notice that the test is not well defined and will raise an error accordingly.
826
+
827
+
Before moving onwards to the derived test, note that the :class:`ContainerBase` class takes the additional argument ``pin_prefix=True``, which locks the prefix of all derived tests to this base test.
828
+
This will allow the retrieval of the sources located in the library by any derived test, regardless of what their containing directory is.
Now from the user's perspective, the only thing to do is to import the above base test and specify the required variables and parameters.
839
+
For consistency with the above example, we set the ``platform`` parameter to use Sarus and Singularity, and we configure the test to run on Piz Daint with the built-in programming environment.
840
+
Hence, the above :class:`ContainerTest` is now reduced to the following:
In a similar fashion, any other user could reuse the above :class:`ContainerBase` class and write the test for their own system with a few lines of code.
Copy file name to clipboardExpand all lines: docs/tutorial_basics.rst
+22-4Lines changed: 22 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -417,7 +417,7 @@ We extend our C++ "Hello, World!" example to print the greetings from multiple t
417
417
:language: cpp
418
418
:lines: 6-
419
419
420
-
This program takes as argument the number of threads it will create and it uses ``std::thread``, which is C++11 addition, meaning that we will need to pass ``-std=c++11`` to our compilers.
420
+
This program takes as argument the number of threads it will create and it uses ``std::thread``, which is a C++11 addition, meaning that we will need to pass ``-std=c++11`` to our compilers.
421
421
Here is the corresponding ReFrame test, where the new concepts introduced are highlighted:
422
422
423
423
.. code-block:: console
@@ -429,12 +429,30 @@ Here is the corresponding ReFrame test, where the new concepts introduced are hi
429
429
:lines: 6-
430
430
:emphasize-lines: 11-13
431
431
432
+
433
+
In order to compile applications using ``std::thread`` with GCC and Clang, the ``-pthread`` option has to be passed to the compiler.
434
+
Since the above option might not be valid for other compilers, we use pipeline hooks to differentiate based on the programming environment as follows:
435
+
436
+
.. code-block:: python
437
+
438
+
@rfm.run_before('compile')
439
+
defset_threading_flags(self):
440
+
environ =self.current_environ.name
441
+
if environ in {'clang', 'gnu'}:
442
+
self.build_system.cxxflags += ['-pthread']
443
+
444
+
445
+
.. note::
446
+
447
+
The pipeline hooks, as well as the regression test pipeline itself, are covered in more detail later on in the tutorial.
448
+
449
+
432
450
ReFrame delegates the compilation of a test to a *build system*, which is an abstraction of the steps needed to compile the test.
433
451
Build systems take also care of interactions with the programming environment if necessary.
434
452
Compilation flags are a property of the build system.
435
453
If not explicitly specified, ReFrame will try to pick the correct build system (e.g., CMake, Autotools etc.) by inspecting the test resources, but in cases as the one presented here where we need to set the compilation flags, we need to specify a build system explicitly.
436
-
In this example, we instruct ReFrame to compile a single source file using the ``-std=c++11 -Wall`` compilation flags.
437
-
Finally, we set the arguments to be passed to the generated executable in :attr:`~reframe.core.pipeline.RegressionTest.executable_opts`.
454
+
In this example, we instruct ReFrame to compile a single source file using the ``-std=c++11 -pthread -Wall`` compilation flags.
455
+
Finally, we set the arguments to be passed to the generated executable in :attr:`executable_opts <reframe.core.pipeline.RegressionTest.executable_opts>`.
438
456
439
457
440
458
.. code-block:: console
@@ -1075,7 +1093,7 @@ Let's see and comment the changes:
1075
1093
First of all, we need to add the new programming environments in the list of the supported ones.
1076
1094
Now there is the problem that each compiler has its own flags for enabling OpenMP, so we need to differentiate the behavior of the test based on the programming environment.
1077
1095
For this reason, we define the flags for each compiler in a separate dictionary (``self.flags``) and we set them in the :func:`setflags` pipeline hook.
1078
-
Let's explain what is this all about.
1096
+
We have first seen the pipeline hooks in the multithreaded "Hello, World!" example and now we explain them in more detail.
1079
1097
When ReFrame loads a test file, it instantiates all the tests it finds in it.
1080
1098
Based on the system ReFrame runs on and the supported environments of the tests, it will generate different test cases for each system partition and environment combination and it will finally send the test cases for execution.
1081
1099
During its execution, a test case goes through the *regression test pipeline*, which is a series of well defined phases.
0 commit comments