Skip to content

Commit 3861980

Browse files
author
Vasileios Karakasis
authored
Merge branch 'master' into feat/dont-restage
2 parents edf2edb + 973d39e commit 3861980

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

64 files changed

+2268
-2460
lines changed

ci-scripts/ci-runner.bash

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -41,8 +41,8 @@ checked_exec()
4141

4242
run_tutorial_checks()
4343
{
44-
cmd="./bin/reframe -C tutorial/config/settings.py \
45-
--save-log-files -r -t tutorial $@"
44+
cmd="./bin/reframe -C tutorials/config/settings.py -J account=jenscscs \
45+
--save-log-files -r -c tutorials/ -R -x HelloThreadedExtendedTest $@"
4646
echo "[INFO] Running tutorial checks with \`$cmd'"
4747
checked_exec $cmd
4848
}
@@ -152,7 +152,7 @@ elif [ $CI_TUTORIAL -eq 1 ]; then
152152
# Run tutorial checks
153153
# Find modified or added tutorial checks
154154
tutorialchecks=( $(git diff origin/master...HEAD --name-only --oneline --no-merges | \
155-
grep -e '^tutorial/.*\.py') )
155+
grep -e '^tutorials/.*\.py') )
156156

157157
if [ ${#tutorialchecks[@]} -ne 0 ]; then
158158
tutorialchecks_path=""

docs/conf.py

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -144,10 +144,6 @@
144144
'logo_only': True
145145
}
146146
html_logo = "_static/img/reframe-logo-dark-bg.svg"
147-
148-
149-
#autodoc_default_flags=['members', 'undoc-members', 'private-members', 'special-members', 'inherited-members', 'show-inheritance']
150-
#autodoc_default_flags=['members', 'undoc-members', 'inherited-members', 'show-inheritance']
151147
autodoc_default_flags = ['members']
152148

153149
# End of READTHEDOCS
@@ -179,7 +175,6 @@
179175
# so a file named "default.css" will overwrite the builtin "default.css".
180176
html_static_path = ['_static']
181177

182-
183178
# Custom sidebar templates, must be a dictionary that maps document names
184179
# to template names.
185180
#

docs/config_reference.rst

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -313,6 +313,9 @@ System Partition Configuration
313313

314314

315315

316+
.. _container-platform-configuration:
317+
318+
316319
Container Platform Configuration
317320
================================
318321

docs/configure.rst

Lines changed: 110 additions & 137 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,7 @@ Configuring ReFrame for Your Site
55
ReFrame comes pre-configured with a minimal generic configuration that will allow you to run ReFrame on any system.
66
This will allow you to run simple local tests using the default compiler of the system.
77
Of course, ReFrame is much more powerful than that.
8-
This section will guide you through configuring ReFrame for your HPC cluster.
9-
We will use as a starting point a simplified configuration for the `Piz Daint <https://www.cscs.ch/computers/piz-daint/>`__ supercomputer at CSCS and we will elaborate along the way.
8+
This section will guide you through configuring ReFrame for your site.
109

1110
If you started using ReFrame from version 3.0, you can keep on reading this section, otherwise you are advised to have a look first at the :doc:`migration_2_to_3` page.
1211

@@ -50,10 +49,10 @@ We'll refer to these top-level properties as *sections*.
5049
These sections contain other objects which further define in detail the framework's behavior.
5150
If you are using a Python file to configure ReFrame, this big JSON configuration object is stored in a special variable called ``site_configuration``.
5251

53-
We will explore the basic configuration of ReFrame through the following configuration file that permits ReFrame to run on Piz Daint.
52+
We will explore the basic configuration of ReFrame by looking into the configuration file of the tutorials, which permits ReFrame to run both on the Piz Daint supercomputer and a local computer.
5453
For the complete listing and description of all configuration options, you should refer to the :doc:`config_reference`.
5554

56-
.. literalinclude:: ../tutorial/config/settings.py
55+
.. literalinclude:: ../tutorials/config/settings.py
5756
:lines: 10-
5857

5958
There are three required sections that each configuration file must provide: ``systems``, ``environments`` and ``logging``.
@@ -68,8 +67,8 @@ ReFrame allows you to configure multiple systems in the same configuration file.
6867
Each system is a different object inside the ``systems`` section.
6968
In our example we define only one system, namely Piz Daint:
7069

71-
.. literalinclude:: ../tutorial/config/settings.py
72-
:lines: 11-75
70+
.. literalinclude:: ../tutorials/config/settings.py
71+
:lines: 11-73
7372

7473
Each system is associated with a set of properties, which in this case are the following:
7574

@@ -89,8 +88,8 @@ In the example shown here, we define three partitions that none of them correspo
8988
The ``login`` partition refers to the login nodes of the system, whereas the ``gpu`` and ``mc`` partitions refer to two different set of nodes in the same cluster that are effectively separated using Slurm constraints.
9089
Let's pick the ``gpu`` partition and look into it in more detail:
9190

92-
.. literalinclude:: ../tutorial/config/settings.py
93-
:lines: 31-51
91+
.. literalinclude:: ../tutorials/config/settings.py
92+
:lines: 31-58
9493

9594
The basic properties of a partition are the following:
9695

@@ -128,31 +127,18 @@ Environments in ReFrame are configured under the ``environments`` section of the
128127
In our configuration example for Piz Daint, we define each ReFrame environment to correspond to each of the Cray-provided programming environments.
129128
In other systems, you could define a ReFrame environment to wrap a toolchain (MPI + compiler combination):
130129

131-
.. literalinclude:: ../tutorial/config/settings.py
132-
:lines: 76-93
130+
.. literalinclude:: ../tutorials/config/settings.py
131+
:lines: 74-125
133132

134133
Each environment is associated with a name.
135134
This name will be used to reference this environment in different contexts, as for example in the ``environs`` property of the system partitions.
136-
This environment definition is minimal, since the default values for the rest of the properties serve our purpose.
135+
A programming environment in ReFrame is essentially a collection of environment modules, environment variables and compiler definitions.
137136

138-
An important feature in ReFrame's configuration, is that you can define section objects differently for different systems or system partitions.
139-
In the following, for demonstration purposes, we define ``PrgEnv-gnu`` differently for the ``mc`` partition of the ``daint`` system (notice the condensed form of writing this as ``daint:mc``):
140-
141-
.. code-block:: python
142-
143-
{
144-
'name': 'PrgEnv-gnu',
145-
'modules': ['PrgEnv-gnu', 'openmpi'],
146-
'cc': 'mpicc',
147-
'cxx': 'mpicxx',
148-
'ftn': 'mpif90',
149-
'target_systems': ['daint:mc']
150-
}
151-
152-
This environment loads different modules and sets the compilers differently, but the most important part is the ``target_systems`` property.
153-
This property is a list of systems or system/partition combinations (as in this case) where this definition of the environment is in effect.
154-
This means that ``PrgEnv-gnu`` will defined this way only for regression tests running on ``daint:mc``.
155-
For all the other systems, it will be defined as shown before.
137+
An important feature in ReFrame's configuration, is that you can define section objects differently for different systems or system partitions by using the ``target_systems`` property.
138+
Notice, for example, how the ``gnu`` environment is defined differently for the system ``daint`` compared to the generic definition.
139+
The ``target_systems`` property is a list of systems or system/partition combinations where this definition of the environment is in effect.
140+
This means that ``gnu`` will be defined this way only for regression tests running on ``daint``.
141+
For all the other systems, it will be defined using the first definition.
156142

157143

158144
---------------------
@@ -163,8 +149,8 @@ ReFrame has a powerful logging mechanism that gives fine grained control over wh
163149
Additionally, it allows for logging performance data from performance tests into different channels.
164150
Let's see how logging is defined in our example configuration, which also represents a typical one for logging:
165151

166-
.. literalinclude:: ../tutorial/config/settings.py
167-
:lines: 94-130
152+
.. literalinclude:: ../tutorials/config/settings.py
153+
:lines: 126-162
168154

169155
Logging is configured under the ``logging`` section of the configuration, which is a list of logger objects.
170156
Unless you want to configure logging differently for different systems, a single logger object is enough.
@@ -250,16 +236,16 @@ To better understand this, let's assume that we have the following ``environment
250236
251237
'environments': [
252238
{
253-
'name': 'PrgEnv-cray',
254-
'modules': ['PrgEnv-cray']
239+
'name': 'cray',
240+
'modules': ['cray']
255241
},
256242
{
257-
'name': 'PrgEnv-gnu',
258-
'modules': ['PrgEnv-gnu']
243+
'name': 'gnu',
244+
'modules': ['gnu']
259245
},
260246
{
261-
'name': 'PrgEnv-gnu',
262-
'modules': ['PrgEnv-gnu', 'openmpi'],
247+
'name': 'gnu',
248+
'modules': ['gnu', 'openmpi'],
263249
'cc': 'mpicc',
264250
'cxx': 'mpicxx',
265251
'ftn': 'mpif90',
@@ -268,7 +254,7 @@ To better understand this, let's assume that we have the following ``environment
268254
],
269255
270256
271-
If the selected system is ``foo``, then ReFrame will use the second definition of ``PrgEnv-gnu`` which is specific to the ``foo`` system.
257+
If the selected system is ``foo``, then ReFrame will use the second definition of ``gnu`` which is specific to the ``foo`` system.
272258

273259
You can override completely the system auto-selection process by specifying a system or system/partition combination with the ``--system`` option, e.g., ``--system=daint`` or ``--system=daint:gpu``.
274260

@@ -284,135 +270,122 @@ Let's see some concrete examples:
284270

285271
* Query the current system's partitions:
286272

287-
.. code::
273+
.. code-block:: console
288274
289-
./bin/reframe -C tutorial/config/settings.py --system=daint --show-config=systems/0/partitions
275+
./bin/reframe -C tutorials/config/settings.py --system=daint --show-config=systems/0/partitions
290276
291277
.. code:: javascript
292278
293-
[
294-
{
295-
"name": "login",
296-
"descr": "Login nodes",
297-
"scheduler": "local",
298-
"launcher": "local",
299-
"environs": [
300-
"PrgEnv-cray",
301-
"PrgEnv-gnu",
302-
"PrgEnv-intel",
303-
"PrgEnv-pgi"
304-
],
305-
"max_jobs": 4
306-
},
307-
{
308-
"name": "gpu",
309-
"descr": "Hybrid nodes (Haswell/P100)",
310-
"scheduler": "slurm",
311-
"launcher": "srun",
312-
"modules": [
313-
"daint-gpu"
314-
],
315-
"access": [
316-
"--constraint=gpu"
317-
],
318-
"environs": [
319-
"PrgEnv-cray",
320-
"PrgEnv-gnu",
321-
"PrgEnv-intel",
322-
"PrgEnv-pgi"
323-
],
324-
"container_platforms": [
325-
{
326-
"name": "Singularity",
327-
"modules": [
328-
"Singularity"
329-
]
330-
}
331-
],
332-
"max_jobs": 100
333-
},
334-
{
335-
"name": "mc",
336-
"descr": "Multicore nodes (Broadwell)",
337-
"scheduler": "slurm",
338-
"launcher": "srun",
339-
"modules": [
340-
"daint-mc"
341-
],
342-
"access": [
343-
"--constraint=mc"
344-
],
345-
"environs": [
346-
"PrgEnv-cray",
347-
"PrgEnv-gnu",
348-
"PrgEnv-intel",
349-
"PrgEnv-pgi"
350-
],
351-
"container_platforms": [
352-
{
353-
"name": "Singularity",
354-
"modules": [
355-
"Singularity"
356-
]
357-
}
358-
],
359-
"max_jobs": 100
360-
}
361-
]
279+
[
280+
{
281+
"name": "login",
282+
"descr": "Login nodes",
283+
"scheduler": "local",
284+
"launcher": "local",
285+
"environs": [
286+
"gnu",
287+
"intel",
288+
"pgi",
289+
"cray"
290+
],
291+
"max_jobs": 10
292+
},
293+
{
294+
"name": "gpu",
295+
"descr": "Hybrid nodes",
296+
"scheduler": "slurm",
297+
"launcher": "srun",
298+
"access": [
299+
"-C gpu",
300+
"-A csstaff"
301+
],
302+
"environs": [
303+
"gnu",
304+
"intel",
305+
"pgi",
306+
"cray"
307+
],
308+
"max_jobs": 100
309+
},
310+
{
311+
"name": "mc",
312+
"descr": "Multicore nodes",
313+
"scheduler": "slurm",
314+
"launcher": "srun",
315+
"access": [
316+
"-C mc",
317+
"-A csstaff"
318+
],
319+
"environs": [
320+
"gnu",
321+
"intel",
322+
"pgi",
323+
"cray"
324+
],
325+
"max_jobs": 100
326+
}
327+
]
362328
363329
Check how the output changes if we explicitly set system to ``daint:login``:
364330

365-
.. code::
331+
.. code-block:: console
366332
367-
./bin/reframe -C tutorial/config/settings.py --system=daint:login --show-config=systems/0/partitions
333+
./bin/reframe -C tutorials/config/settings.py --system=daint:login --show-config=systems/0/partitions
368334
369335
370336
.. code:: javascript
371337
372-
[
373-
{
374-
"name": "login",
375-
"descr": "Login nodes",
376-
"scheduler": "local",
377-
"launcher": "local",
378-
"environs": [
379-
"PrgEnv-cray",
380-
"PrgEnv-gnu",
381-
"PrgEnv-intel",
382-
"PrgEnv-pgi"
383-
],
384-
"max_jobs": 4
385-
}
386-
]
338+
[
339+
{
340+
"name": "login",
341+
"descr": "Login nodes",
342+
"scheduler": "local",
343+
"launcher": "local",
344+
"environs": [
345+
"gnu",
346+
"intel",
347+
"pgi",
348+
"cray"
349+
],
350+
"max_jobs": 10
351+
}
352+
]
353+
387354
388355
ReFrame will internally represent system ``daint`` as having a single partition only.
389356
Notice also how you can use indexes to objects elements inside a list.
390357

391358
* Query an environment configuration:
392359

393-
.. code::
360+
.. code-block:: console
394361
395-
./bin/reframe -C tutorial/config/settings.py --system=daint --show-config=environments/@PrgEnv-gnu
362+
./bin/reframe -C tutorials/config/settings.py --system=daint --show-config=environments/@gnu
396363
397364
.. code:: javascript
398365
399-
{
400-
"name": "PrgEnv-gnu",
401-
"modules": [
402-
"PrgEnv-gnu"
403-
]
404-
}
366+
{
367+
"name": "gnu",
368+
"modules": [
369+
"PrgEnv-gnu"
370+
],
371+
"cc": "cc",
372+
"cxx": "CC",
373+
"ftn": "ftn",
374+
"target_systems": [
375+
"daint"
376+
]
377+
}
405378
406379
If an object has a ``name`` property you can address it by name using the ``@name`` syntax, instead of its index.
407380

408381
* Query an environment's compiler:
409382

410-
.. code::
383+
.. code-block:: console
411384
412-
./bin/reframe -C tutorial/config/settings.py --system=daint --show-config=environments/@PrgEnv-gnu/cxx
385+
./bin/reframe -C tutorials/config/settings.py --system=daint --show-config=environments/@gnu/cxx
413386
414387
.. code:: javascript
415388
416389
"CC"
417390
418-
Notice that although the C++ compiler is not defined in the environment's definitions, ReFrame will print the default value, if you explicitly query its value.
391+
If you explicitly query a configuration value which is not defined in the configuration file, ReFrame will print its default value.

0 commit comments

Comments
 (0)