Merge branch 'master' into slurm_cpus_per_task_bugfix

Author: vkarak

docs/config_reference.rst

@@ -142,7 +142,7 @@ System Configuration
    These modules modify the ReFrame environment.
    This is useful in cases where a particular module is needed, for example, to submit jobs on a specific system.
 
-.. js:attribute:: .systems[].variables
+.. js:attribute:: .systems[].env_vars
 
    :required: No
    :default: ``[]``
@@ -154,6 +154,14 @@ System Configuration
    ReFrame will expand its value.
    Variables are set after the environment modules are loaded.
 
+   .. versionadded:: 4.0.0
+
+.. js:attribute:: .systems[].variables
+
+   .. deprecated:: 4.0.0
+      Please use :js:attr:`env_vars` instead.
+      If specified in conjunction with :js:attr:`env_vars`, it will be ignored.
+
 .. js:attribute:: .systems[].prefix
 
    :required: No
@@ -374,7 +382,7 @@ System Partition Configuration
    When the value is ``null``, no time limit is applied.
 
 
-.. js:attribute:: .systems[].partitions[].variables
+.. js:attribute:: .systems[].partitions[].env_vars
 
    :required: No
    :default: ``[]``
@@ -385,6 +393,13 @@ System Partition Configuration
    ReFrame will expand its value.
    Variables are set after the environment modules are loaded.
 
+   .. versionadded:: 4.0.0
+
+.. js:attribute:: .systems[].partitions[].variables
+
+   .. deprecated:: 4.0.0
+      Please use :js:attr:`env_vars` instead.
+      If specified in conjunction with :js:attr:`env_vars`, it will be ignored.
 
 .. js:attribute:: .systems[].partitions[].max_jobs
 
@@ -500,7 +515,7 @@ ReFrame can launch containerized applications, but you need to configure properl
    A list of `environment module objects <#module-objects>`__ to be loaded when running containerized tests using this container platform.
 
 
-.. js:attribute:: .systems[].partitions[].container_platforms[].variables
+.. js:attribute:: .systems[].partitions[].container_platforms[].env_vars
 
    :required: No
    :default: ``[]``
@@ -511,6 +526,14 @@ ReFrame can launch containerized applications, but you need to configure properl
    ReFrame will expand its value.
    Variables are set after the environment modules are loaded.
 
+   .. versionadded:: 4.0.0
+
+.. js:attribute:: .systems[].partitions[].container_platforms[].variables
+
+   .. deprecated:: 4.0.0
+      Please use :js:attr:`env_vars` instead.
+      If specified in conjunction with :js:attr:`env_vars`, it will be ignored.
+
 
 Custom Job Scheduler Resources
 ==============================
@@ -619,7 +642,7 @@ They are associated with `system partitions <#system-partition-configuration>`__
    A list of `environment module objects <#module-objects>`__ to be loaded when this environment is loaded.
 
 
-.. js:attribute:: .environments[].variables
+.. js:attribute:: .environments[].env_vars
 
    :required: No
    :default: ``[]``
@@ -630,6 +653,14 @@ They are associated with `system partitions <#system-partition-configuration>`__
    ReFrame will expand its value.
    Variables are set after the environment modules are loaded.
 
+   .. versionadded:: 4.0.0
+
+.. js:attribute:: .environments[].variables
+
+   .. deprecated:: 4.0.0
+      Please use :js:attr:`env_vars` instead.
+      If specified in conjunction with :js:attr:`env_vars`, it will be ignored.
+
 
 .. js:attribute:: .environments[].features
 
@@ -868,7 +899,8 @@ All logging handlers share the following set of common attributes:
       ``%(check_descr)s``, The value of the :attr:`~reframe.core.pipeline.RegressionTest.descr` attribute.
       ``%(check_display_name)s``, The value of the :attr:`~reframe.core.pipeline.RegressionTest.display_name` attribute.
       ``%(check_environ)s``, The name of the test's :attr:`~reframe.core.pipeline.RegressionTest.current_environ`.
-      ``%(check_exclusive_access)s``, The value of the :attr:`~reframe.core.pipeline.RegressionTest.exclusive_access` attribute.
+      ``%(check_env_vars)s``, The value of the :attr:`~reframe.core.pipeline.RegressionTest.env_vars` attribute.
+    ``%(check_exclusive_access)s``, The value of the :attr:`~reframe.core.pipeline.RegressionTest.exclusive_access` attribute.
       ``%(check_executable)s``, The value of the :attr:`~reframe.core.pipeline.RegressionTest.executable` attribute.
       ``%(check_executable_opts)s``, The value of the :attr:`~reframe.core.pipeline.RegressionTest.executable_opts` attribute.
       ``%(check_extra_resources)s``, The value of the :attr:`~reframe.core.pipeline.RegressionTest.extra_resources` attribute.
@@ -918,7 +950,7 @@ All logging handlers share the following set of common attributes:
       ``%(check_use_multithreading)s``, The value of the :attr:`~reframe.core.pipeline.RegressionTest.use_multithreading` attribute.
       ``%(check_valid_prog_environs)s``, The value of the :attr:`~reframe.core.pipeline.RegressionTest.valid_prog_environs` attribute.
       ``%(check_valid_systems)s``, The value of the :attr:`~reframe.core.pipeline.RegressionTest.valid_systems` attribute.
-      ``%(check_variables)s``, The value of the :attr:`~reframe.core.pipeline.RegressionTest.variables` attribute.
+      ``%(check_variables)s``, DEPRECATED: Please use ``%(check_env_vars)s`` instead.
       ``%(osuser)s``, The name of the OS user running ReFrame.
       ``%(osgroup)s``, The name of the OS group running ReFrame.
       ``%(version)s``, The ReFrame version.

docs/tutorial_basics.rst

@@ -430,7 +430,7 @@ The next thing to notice is the :attr:`~reframe.core.pipeline.RegressionTest.pre
 These commands will be executed from the test's stage directory.
 In this case, we just fetch the source code of the benchmark.
 For running the benchmark, we need to set the OpenMP number of threads and pin them to the right CPUs through the ``OMP_NUM_THREADS`` and ``OMP_PLACES`` environment variables.
-You can set environment variables in a ReFrame test through the :attr:`~reframe.core.pipeline.RegressionTest.variables` dictionary.
+You can set environment variables in a ReFrame test through the :attr:`~reframe.core.pipeline.RegressionTest.env_vars` dictionary.
 
 What makes a ReFrame test a performance test is the definition of at least one :ref:`performance function<deferrable-performance-functions>`.
 Similarly to a test's :func:`@sanity_function<reframe.core.pipeline.RegressionMixin.sanity_function>`, a performance function is a member function decorated with the :attr:`@performance_function<reframe.core.pipeline.RegressionMixin.performance_function>` decorator, which binds the decorated function to a given unit.

reframe/core/config.py

@@ -347,6 +347,45 @@ def validate(self):
 
                 partition_names.add(partname)
 
+        def _warn_variables(config, opt_path):
+            opt_path = '/'.join(opt_path + ['variables'])
+            if 'env_vars' in config and 'variables' in config:
+                getlogger().warning(
+                    f"configuration option {opt_path!r}: "
+                    f"both 'env_vars' and 'variables' are defined; "
+                    f"'variables' will be ignored"
+                )
+            elif 'variables' in config:
+                getlogger().warning(
+                    f"configuration option {opt_path!r}: "
+                    f"'variables' is deprecated; please use 'env_vars' instead"
+                )
+                config['env_vars'] = config['variables']
+
+        # Warn about the deprecated `variables` and convert them internally to
+        # `env_vars`
+        for system in self._site_config['systems']:
+            sysname = system['name']
+            opt_path = ['systems', f'@{sysname}']
+            _warn_variables(system, opt_path)
+            for part in system['partitions']:
+                partname = part['name']
+                opt_path += ['partitions', f'@{partname}']
+                _warn_variables(part, opt_path)
+                for i, cp in enumerate(part.get('container_platforms', [])):
+                    opt_path += ['container_platforms', str(i)]
+                    _warn_variables(cp, opt_path)
+                    opt_path.pop()
+                    opt_path.pop()
+
+                opt_path.pop()
+                opt_path.pop()
+
+        for env in self._site_config['environments']:
+            envname = env['name']
+            opt_path = ['environments', f'@{envname}']
+            _warn_variables(env, opt_path)
+
     def select_subconfig(self, system_fullname=None,
                          ignore_resolve_errors=False):
         # First look for the current subconfig in the cache; if not found,

reframe/core/environments.py

@@ -10,6 +10,7 @@
 import reframe.utility as util
 import reframe.utility.jsonext as jsonext
 import reframe.utility.typecheck as typ
+from reframe.core.warnings import user_deprecation_warning
 
 
 def normalize_module_list(modules):
@@ -37,14 +38,22 @@ class Environment(jsonext.JSONSerializable):
        Users may not create :class:`Environment` objects directly.
     '''
 
-    def __init__(self, name, modules=None, variables=None,
+    def __init__(self, name, modules=None, env_vars=None,
                  extras=None, features=None):
         modules = modules or []
-        variables = variables or []
+        env_vars = env_vars or []
         self._name = name
         self._modules = normalize_module_list(modules)
         self._module_names = [m['name'] for m in self._modules]
-        self._variables = collections.OrderedDict(variables)
+
+        # Convert values of env_vars to strings before storing
+        if isinstance(env_vars, dict):
+            env_vars = env_vars.items()
+
+        self._env_vars = collections.OrderedDict()
+        for k, v in env_vars:
+            self._env_vars[k] = str(v)
+
         self._extras = extras or {}
         self._features = features or []
 
@@ -85,12 +94,25 @@ def modules_detailed(self):
         return util.SequenceView(self._modules)
 
     @property
-    def variables(self):
+    def env_vars(self):
         '''The environment variables associated with this environment.
 
         :type: :class:`OrderedDict[str, str]`
+
+        .. versionadded:: 4.0.0
+        '''
+        return util.MappingView(self._env_vars)
+
+    @property
+    def variables(self):
+        '''The environment variables associated with this environment.
+
+        .. deprecated:: 4.0.0
+           Please :attr:`env_vars` instead.
         '''
-        return util.MappingView(self._variables)
+        user_deprecation_warning("the 'variables' attribute is deprecated; "
+                                 "please use the 'env_vars' instead")
+        return util.MappingView(self._env_vars)
 
     @property
     def extras(self):
@@ -117,9 +139,19 @@ def __eq__(self, other):
         if not isinstance(other, type(self)):
             return NotImplemented
 
-        return (self.name == other.name and
-                set(self.modules) == set(other.modules) and
-                self.variables == other.variables)
+        if (self.name != other.name or
+            set(self.modules) != set(other.modules)):
+            return False
+
+        # Env. variables are checked against their string representation
+        for kv0, kv1 in zip(self.env_vars.items(),
+                            other.env_vars.items()):
+            k0, v0 = kv0
+            k1, v1 = kv1
+            if k0 != k1 or str(v0) != str(v1):
+                return False
+
+        return True
 
     def __str__(self):
         return self.name
@@ -128,7 +160,7 @@ def __repr__(self):
         return (f'{type(self).__name__}('
                 f'name={self._name!r}, '
                 f'modules={self._modules!r}, '
-                f'variables={list(self._variables.items())!r}, '
+                f'env_vars={list(self._env_vars.items())!r}, '
                 f'extras={self._extras!r}, features={self._features!r})')
 
 
@@ -141,15 +173,15 @@ def __init__(self, name='env_snapshot'):
     def restore(self):
         '''Restore this environment snapshot.'''
         os.environ.clear()
-        os.environ.update(self._variables)
+        os.environ.update(self._env_vars)
 
     def __eq__(self, other):
         if not isinstance(other, Environment):
             return NotImplemented
 
-        # Order of variables is not important when comparing snapshots
-        for k, v in self.variables.items():
-            if other.variables[k] != v:
+        # Order of env. variables is not important when comparing snapshots
+        for k, v in self.env_vars.items():
+            if other.env_vars[k] != v:
                 return False
 
         return self.name == other.name
@@ -185,7 +217,7 @@ class ProgEnvironment(Environment):
     def __init__(self,
                  name,
                  modules=None,
-                 variables=None,
+                 env_vars=None,
                  extras=None,
                  features=None,
                  cc='cc',
@@ -198,7 +230,7 @@ def __init__(self,
                  fflags=None,
                  ldflags=None,
                  **kwargs):
-        super().__init__(name, modules, variables, extras, features)
+        super().__init__(name, modules, env_vars, extras, features)
         self._cc = cc
         self._cxx = cxx
         self._ftn = ftn

reframe/core/logging.py

@@ -20,6 +20,7 @@
 import reframe.utility.jsonext as jsonext
 import reframe.utility.osext as osext
 from reframe.core.exceptions import ConfigError, LoggingError
+from reframe.core.warnings import suppress_deprecations
 from reframe.utility.profile import TimeProfiler
 
 
@@ -396,8 +397,8 @@ def __init__(self, url, extras=None, ignore_keys=None):
     def _record_to_json(self, record):
         def _can_send(key):
             return not (
-                key.startswith('_') or key in HTTPJSONHandler.LOG_ATTRS
-                or (self._ignore_keys and key in self._ignore_keys)
+                key.startswith('_') or key in HTTPJSONHandler.LOG_ATTRS or
+                (self._ignore_keys and key in self._ignore_keys)
             )
 
         json_record = {
@@ -577,9 +578,11 @@ def _update_check_extras(self):
         for attr, alt_name in check_type.loggable_attrs():
             extra_name  = alt_name or attr
 
-            # In case of AttributeError, i.e., the variable is undefined, we
-            # set the value to None
-            val = getattr(self.check, attr, None)
+            with suppress_deprecations():
+                # In case of AttributeError, i.e., the variable is undefined,
+                # we set the value to None
+                val = getattr(self.check, attr, None)
+
             if attr in check_type.raw_params:
                 # Attribute is parameter, so format it
                 val = check_type.raw_params[attr].format(val)

reframe/core/pipeline.py

@@ -781,11 +781,28 @@ def pipeline_hooks(cls):
 
     #: Environment variables to be set before running this test.
     #:
-    #: These variables will be set during the :func:`setup` phase.
+    #: The value of the environment variables can be of any type. ReFrame will
+    #: invoke :func:`str` on it whenever it needs to emit it in a script.
     #:
-    #: :type: :class:`Dict[str, str]`
+    #: :type: :class:`Dict[str, object]`
     #: :default: ``{}``
-    variables = variable(typ.Dict[str, str], value={}, loggable=True)
+    #:
+    #: .. versionadded:: 4.0.0
+    env_vars = variable(typ.Dict[str, str],
+                        typ.Dict[str, object], value={}, loggable=True)
+    # NOTE: We still keep the original type, just to allow setting this
+    # variable from the command line, because otherwise, ReFrame will not know
+    # how to convert a value to an arbitrary object.
+
+    #: Environment variables to be set before running this test.
+    #:
+    #: This is an alias of :attr:`env_vars`.
+    #:
+    #: .. deprecated:: 4.0.0
+    #:    Please use :attr:`env_vars` instead.
+    variables = deprecate(variable(alias=env_vars, loggable=True),
+                          f"the use of 'variables' is deprecated; "
+                          f"please use 'env_vars' instead")
 
     #: Time limit for this test.
     #:
@@ -1746,7 +1763,7 @@ def compile(self):
             self.build_system.executable = self.executable
 
         user_environ = Environment(self.unique_name,
-                                   self.modules, self.variables.items())
+                                   self.modules, self.env_vars.items())
         environs = [self._current_partition.local_env, self._current_environ,
                     user_environ, self._cdt_environ]
         self._build_job.time_limit = (
@@ -1880,7 +1897,7 @@ def _get_cp_env():
             *self.postrun_cmds
         ]
         user_environ = Environment(self.unique_name,
-                                   self.modules, self.variables.items())
+                                   self.modules, self.env_vars.items())
         environs = [
             self._current_partition.local_env,
             self._current_environ,