Merge branch 'master' into feat/slurm-N-option

Vasileios Karakasis · web-flow · commit 59718551f6ef · 2020-06-03T20:59:55.000+02:00
diff --git a/docs/dependencies.rst b/docs/dependencies.rst
@@ -170,3 +170,14 @@ In fact, you can rewrite :func:`set_executable` function as follows:
 Now it's easier to understand what the ``@require_deps`` decorator does behind the scenes.
 It binds the function arguments to a partial realization of the :func:`getdep` function and attaches the decorated function as an after-setup hook.
 In fact, any ``@require_deps``-decorated function will be invoked before any other after-setup hook.
+
+
+.. _cleaning-up-stage-files:
+
+Cleaning up stage files
+-----------------------
+
+In principle, the output of a test might be needed by its dependent tests.
+As a result, the stage directory of the test will only be cleaned up after all of its *immediate* dependent tests have finished successfully.
+If any of its children has failed, the cleanup phase will be skipped, such that all the test's files will remain in the stage directory.
+This allows users to reproduce manually the error of a failed test with dependencies, since all the needed resources of the failing test are left in their original location.
diff --git a/docs/manpage.rst b/docs/manpage.rst
@@ -366,15 +366,6 @@ If no node can be selected, the test will be marked as a failure with an appropr
      This is the default policy.
    - Any positive integer: Flexible tests will be assigned as many tasks as needed in order to span over the specified number of nodes from the node pool.
 
-.. option:: --flex-alloc-tasks[=POLICY]
-
-   .. deprecated:: 2.21
-
-      Please use |--flex-alloc-nodes|_ instead.
-
-.. |--flex-alloc-nodes| replace:: :attr:`--flex-alloc-nodes`
-.. _--flex-alloc-nodes: #cmdoption-flex-alloc-nodes
-
 ---------------------------------------
 Options controlling ReFrame environment
 ---------------------------------------
diff --git a/docs/pipeline.rst b/docs/pipeline.rst
@@ -91,6 +91,9 @@ The Cleanup Phase
 During this final stage of the pipeline, the test's resources are cleaned up.
 More specifically, if the test has finished successfully, all interesting test files (build/job scripts, build/job script output and any user-specified files) are copied to ReFrame's output directory and the stage directory of the test is deleted.
 
+.. note::
+   This phase might be deferred in case a test has dependents (see :ref:`cleaning-up-stage-files` for more details).
+
 
 Execution Policies
 ------------------
@@ -129,3 +132,19 @@ ReFrame tries to keep concurrency high by maintaining as many test cases as poss
 When the `concurrency limit <config_reference.html#.systems[].partitions[].max_jobs>`__ is reached, ReFrame will first try to free up execution slots by checking if any of the spawned jobs have finished, and it will fill that slots first before throttling execution.
 
 ReFrame uses polling to check the status of the spawned jobs, but it does so in a dynamic way, in order to ensure both responsiveness and avoid overloading the system job scheduler with excessive polling.
+
+Timing the Test Pipeline
+------------------------
+
+.. versionadded:: 3.0
+
+ReFrame keeps track of the time a test spends in every pipeline stage and reports that after each test finishes.
+However, it does so from its own perspective and not from that of the scheduler backend used.
+This has some practical implications:
+As soon as a test enters the "run" phase, ReFrame's timer for that phase starts ticking regardless if the associated job is pending.
+Similarly, the "run" phase ends as soon as ReFrame realizes it.
+This will happen after the associated job has finished.
+For this reason, the time spent in the pipeline's "run" phase should *not* be interpreted as the actual runtime of the test, especially if a non-local scheduler backend is used.
+
+Finally, the execution time of the "cleanup" phase is not reported when a test finishes, since it may be deferred in case that there exist tests that depend on that one.
+See :doc:`dependencies` for more information on how ReFrame treats tests with dependencies.
diff --git a/docs/tutorial_basic.rst b/docs/tutorial_basic.rst
@@ -177,50 +177,51 @@ If everything is configured correctly for your system, you should get an output
 
 .. code-block:: none
 
-   [ReFrame Setup]
-     version:           3.0-dev6 (rev: 89d50861)
-     command:           './bin/reframe -C tutorial/config/settings.py -c tutorial/example1.py -r'
-     launched by:       user@daint101
-     working directory: '/path/to/reframe'
-     check search path: (R) '/path/to/reframe/tutorial/example1.py'
-     stage directory:   '/path/to/reframe/stage'
-     output directory:  '/path/to/reframe/output'
-
-   [==========] Running 1 check(s)
-   [==========] Started on Sat May  9 22:10:16 2020
-
-   [----------] started processing Example1Test (Simple matrix-vector multiplication example)
-   [ RUN      ] Example1Test on daint:login using PrgEnv-cray
-   [ RUN      ] Example1Test on daint:login using PrgEnv-gnu
-   [ RUN      ] Example1Test on daint:login using PrgEnv-intel
-   [ RUN      ] Example1Test on daint:login using PrgEnv-pgi
-   [ RUN      ] Example1Test on daint:gpu using PrgEnv-cray
-   [ RUN      ] Example1Test on daint:gpu using PrgEnv-gnu
-   [ RUN      ] Example1Test on daint:gpu using PrgEnv-intel
-   [ RUN      ] Example1Test on daint:gpu using PrgEnv-pgi
-   [ RUN      ] Example1Test on daint:mc using PrgEnv-cray
-   [ RUN      ] Example1Test on daint:mc using PrgEnv-gnu
-   [ RUN      ] Example1Test on daint:mc using PrgEnv-intel
-   [ RUN      ] Example1Test on daint:mc using PrgEnv-pgi
-   [----------] finished processing Example1Test (Simple matrix-vector multiplication example)
-
-   [----------] waiting for spawned checks to finish
-   [       OK ] ( 1/12) Example1Test on daint:mc using PrgEnv-cray
-   [       OK ] ( 2/12) Example1Test on daint:gpu using PrgEnv-intel
-   [       OK ] ( 3/12) Example1Test on daint:gpu using PrgEnv-cray
-   [       OK ] ( 4/12) Example1Test on daint:login using PrgEnv-intel
-   [       OK ] ( 5/12) Example1Test on daint:login using PrgEnv-cray
-   [       OK ] ( 6/12) Example1Test on daint:mc using PrgEnv-gnu
-   [       OK ] ( 7/12) Example1Test on daint:gpu using PrgEnv-gnu
-   [       OK ] ( 8/12) Example1Test on daint:login using PrgEnv-gnu
-   [       OK ] ( 9/12) Example1Test on daint:login using PrgEnv-pgi
-   [       OK ] (10/12) Example1Test on daint:gpu using PrgEnv-pgi
-   [       OK ] (11/12) Example1Test on daint:mc using PrgEnv-intel
-   [       OK ] (12/12) Example1Test on daint:mc using PrgEnv-pgi
-   [----------] all spawned checks have finished
-
-   [  PASSED  ] Ran 12 test case(s) from 1 check(s) (0 failure(s))
-   [==========] Finished on Sat May  9 22:10:46 2020
+  [ReFrame Setup]
+    version:           3.0-dev7 (rev: 85ca676f)
+    command:           './bin/reframe -C tutorial/config/settings.py -c tutorial/example1.py -r'
+    launched by:       user@daint104
+    working directory: '/path/to/reframe'
+    settings file:     'tutorial/config/settings.py'
+    check search path: (R) '/path/to/reframe/tutorial/example1.py'
+    stage directory:   '/path/to/reframe/stage'
+    output directory:  '/path/to/reframe/output'
+
+  [==========] Running 1 check(s)
+  [==========] Started on Wed Jun  3 08:50:46 2020
+
+  [----------] started processing Example1Test (Simple matrix-vector multiplication example)
+  [ RUN      ] Example1Test on daint:login using PrgEnv-cray
+  [ RUN      ] Example1Test on daint:login using PrgEnv-gnu
+  [ RUN      ] Example1Test on daint:login using PrgEnv-intel
+  [ RUN      ] Example1Test on daint:login using PrgEnv-pgi
+  [ RUN      ] Example1Test on daint:gpu using PrgEnv-cray
+  [ RUN      ] Example1Test on daint:gpu using PrgEnv-gnu
+  [ RUN      ] Example1Test on daint:gpu using PrgEnv-intel
+  [ RUN      ] Example1Test on daint:gpu using PrgEnv-pgi
+  [ RUN      ] Example1Test on daint:mc using PrgEnv-cray
+  [ RUN      ] Example1Test on daint:mc using PrgEnv-gnu
+  [ RUN      ] Example1Test on daint:mc using PrgEnv-intel
+  [ RUN      ] Example1Test on daint:mc using PrgEnv-pgi
+  [----------] finished processing Example1Test (Simple matrix-vector multiplication example)
+
+  [----------] waiting for spawned checks to finish
+  [       OK ] ( 1/12) Example1Test on daint:login using PrgEnv-intel [compile: 1.940s run: 20.747s total: 23.778s]
+  [       OK ] ( 2/12) Example1Test on daint:login using PrgEnv-cray [compile: 0.347s run: 25.096s total: 26.591s]
+  [       OK ] ( 3/12) Example1Test on daint:mc using PrgEnv-intel [compile: 2.019s run: 6.286s total: 8.357s]
+  [       OK ] ( 4/12) Example1Test on daint:mc using PrgEnv-cray [compile: 0.506s run: 11.056s total: 11.744s]
+  [       OK ] ( 5/12) Example1Test on daint:gpu using PrgEnv-cray [compile: 0.435s run: 19.499s total: 20.483s]
+  [       OK ] ( 6/12) Example1Test on daint:login using PrgEnv-gnu [compile: 1.648s run: 25.631s total: 27.964s]
+  [       OK ] ( 7/12) Example1Test on daint:mc using PrgEnv-gnu [compile: 1.825s run: 10.434s total: 12.301s]
+  [       OK ] ( 8/12) Example1Test on daint:gpu using PrgEnv-intel [compile: 2.018s run: 16.316s total: 18.529s]
+  [       OK ] ( 9/12) Example1Test on daint:login using PrgEnv-pgi [compile: 1.643s run: 22.118s total: 24.090s]
+  [       OK ] (10/12) Example1Test on daint:gpu using PrgEnv-gnu [compile: 1.729s run: 20.028s total: 21.901s]
+  [       OK ] (11/12) Example1Test on daint:gpu using PrgEnv-pgi [compile: 1.753s run: 15.128s total: 16.923s]
+  [       OK ] (12/12) Example1Test on daint:mc using PrgEnv-pgi [compile: 1.732s run: 35.556s total: 37.330s]
+  [----------] all spawned checks have finished
+
+  [  PASSED  ] Ran 12 test case(s) from 1 check(s) (0 failure(s))
+  [==========] Finished on Wed Jun  3 08:51:46 2020
 
 
 Notice how our regression test is run on every partition of the configured system and for every programming environment.
diff --git a/docs/tutorial_deps.rst b/docs/tutorial_deps.rst
@@ -77,7 +77,7 @@ Here is the output when running the OSU tests with the asynchronous execution po
 .. code-block:: none
 
   [==========] Running 7 check(s)
-  [==========] Started on Wed Mar 25 13:51:06 2020
+  [==========] Started on Wed Jun  3 09:00:40 2020
 
   [----------] started processing OSUBuildTest (OSU benchmarks build test)
   [ RUN      ] OSUBuildTest on daint:gpu using PrgEnv-gnu
@@ -140,31 +140,31 @@ Here is the output when running the OSU tests with the asynchronous execution po
   [----------] finished processing OSUAllreduceTest_16 (OSU Allreduce test)
 
   [----------] waiting for spawned checks to finish
-  [       OK ] ( 1/21) OSUBuildTest on daint:gpu using PrgEnv-pgi
-  [       OK ] ( 2/21) OSUBuildTest on daint:gpu using PrgEnv-gnu
-  [       OK ] ( 3/21) OSUBuildTest on daint:gpu using PrgEnv-intel
-  [       OK ] ( 4/21) OSUAllreduceTest_2 on daint:gpu using PrgEnv-pgi
-  [       OK ] ( 5/21) OSUAllreduceTest_4 on daint:gpu using PrgEnv-pgi
-  [       OK ] ( 6/21) OSUAllreduceTest_8 on daint:gpu using PrgEnv-pgi
-  [       OK ] ( 7/21) OSUAllreduceTest_16 on daint:gpu using PrgEnv-pgi
-  [       OK ] ( 8/21) OSUAllreduceTest_4 on daint:gpu using PrgEnv-gnu
-  [       OK ] ( 9/21) OSUAllreduceTest_16 on daint:gpu using PrgEnv-gnu
-  [       OK ] (10/21) OSUAllreduceTest_8 on daint:gpu using PrgEnv-gnu
-  [       OK ] (11/21) OSUAllreduceTest_16 on daint:gpu using PrgEnv-intel
-  [       OK ] (12/21) OSULatencyTest on daint:gpu using PrgEnv-pgi
-  [       OK ] (13/21) OSUAllreduceTest_2 on daint:gpu using PrgEnv-gnu
-  [       OK ] (14/21) OSULatencyTest on daint:gpu using PrgEnv-gnu
-  [       OK ] (15/21) OSUBandwidthTest on daint:gpu using PrgEnv-pgi
-  [       OK ] (16/21) OSUBandwidthTest on daint:gpu using PrgEnv-gnu
-  [       OK ] (17/21) OSUAllreduceTest_8 on daint:gpu using PrgEnv-intel
-  [       OK ] (18/21) OSUAllreduceTest_4 on daint:gpu using PrgEnv-intel
-  [       OK ] (19/21) OSULatencyTest on daint:gpu using PrgEnv-intel
-  [       OK ] (20/21) OSUAllreduceTest_2 on daint:gpu using PrgEnv-intel
-  [       OK ] (21/21) OSUBandwidthTest on daint:gpu using PrgEnv-intel
+  [       OK ] ( 1/21) OSUBuildTest on daint:gpu using PrgEnv-pgi [compile: 29.581s run: 0.086s total: 29.708s]
+  [       OK ] ( 2/21) OSUBuildTest on daint:gpu using PrgEnv-gnu [compile: 26.250s run: 69.120s total: 95.437s]
+  [       OK ] ( 3/21) OSUBuildTest on daint:gpu using PrgEnv-intel [compile: 39.385s run: 89.213s total: 129.871s]
+  [       OK ] ( 4/21) OSULatencyTest on daint:gpu using PrgEnv-pgi [compile: 0.012s run: 145.355s total: 154.504s]
+  [       OK ] ( 5/21) OSUAllreduceTest_2 on daint:gpu using PrgEnv-pgi [compile: 0.014s run: 148.276s total: 154.433s]
+  [       OK ] ( 6/21) OSUAllreduceTest_4 on daint:gpu using PrgEnv-pgi [compile: 0.011s run: 149.763s total: 154.407s]
+  [       OK ] ( 7/21) OSUAllreduceTest_8 on daint:gpu using PrgEnv-pgi [compile: 0.013s run: 151.262s total: 154.378s]
+  [       OK ] ( 8/21) OSUAllreduceTest_16 on daint:gpu using PrgEnv-pgi [compile: 0.010s run: 152.716s total: 154.360s]
+  [       OK ] ( 9/21) OSULatencyTest on daint:gpu using PrgEnv-gnu [compile: 0.014s run: 210.952s total: 220.847s]
+  [       OK ] (10/21) OSUBandwidthTest on daint:gpu using PrgEnv-pgi [compile: 0.015s run: 213.285s total: 220.758s]
+  [       OK ] (11/21) OSUAllreduceTest_4 on daint:gpu using PrgEnv-gnu [compile: 0.011s run: 215.596s total: 220.717s]
+  [       OK ] (12/21) OSUAllreduceTest_16 on daint:gpu using PrgEnv-gnu [compile: 0.011s run: 218.742s total: 220.651s]
+  [       OK ] (13/21) OSUAllreduceTest_2 on daint:gpu using PrgEnv-intel [compile: 0.013s run: 203.214s total: 206.115s]
+  [       OK ] (14/21) OSUAllreduceTest_8 on daint:gpu using PrgEnv-intel [compile: 0.016s run: 204.819s total: 206.078s]
+  [       OK ] (15/21) OSUBandwidthTest on daint:gpu using PrgEnv-gnu [compile: 0.012s run: 258.772s total: 266.873s]
+  [       OK ] (16/21) OSUAllreduceTest_8 on daint:gpu using PrgEnv-gnu [compile: 0.014s run: 263.576s total: 266.752s]
+  [       OK ] (17/21) OSULatencyTest on daint:gpu using PrgEnv-intel [compile: 0.011s run: 227.234s total: 231.789s]
+  [       OK ] (18/21) OSUAllreduceTest_4 on daint:gpu using PrgEnv-intel [compile: 0.013s run: 229.729s total: 231.724s]
+  [       OK ] (19/21) OSUAllreduceTest_2 on daint:gpu using PrgEnv-gnu [compile: 0.013s run: 286.203s total: 292.444s]
+  [       OK ] (20/21) OSUAllreduceTest_16 on daint:gpu using PrgEnv-intel [compile: 0.028s run: 242.030s total: 242.091s]
+  [       OK ] (21/21) OSUBandwidthTest on daint:gpu using PrgEnv-intel [compile: 0.013s run: 243.719s total: 247.384s]
   [----------] all spawned checks have finished
 
   [  PASSED  ] Ran 21 test case(s) from 7 check(s) (0 failure(s))
-  [==========] Finished on Wed Mar 25 14:37:53 2020
+  [==========] Finished on Wed Jun  3 09:07:24 2020
 
 Before starting running the tests, ReFrame topologically sorts them based on their dependencies and schedules them for running using the selected execution policy.
 With the serial execution policy, ReFrame simply executes the tests to completion as they "arrive", since the tests are already topologically sorted.
diff --git a/reframe/core/deferrable.py b/reframe/core/deferrable.py
@@ -338,20 +338,3 @@ def __abs__(a):
     @deferrable
     def __invert__(a):
         return ~a
-
-
-def evaluate(expr):
-    user_deprecation_warning('evaluate() is deprecated: '
-                             'please use reframe.utility.sanity.evaluate')
-
-    if isinstance(expr, _DeferredExpression):
-        return expr.evaluate()
-    else:
-        return expr
-
-
-@deferrable
-def make_deferrable(a):
-    user_deprecation_warning('make_deferrable() is deprecated: '
-                             'please use reframe.utility.sanity.defer')
-    return a
diff --git a/reframe/core/schedulers/slurm.py b/reframe/core/schedulers/slurm.py
@@ -134,7 +134,14 @@ def completion_time(self, job):
         if not state_match:
             return None
 
-        self._completion_time = max(float(s.group('end')) for s in state_match)
+        completion_times = []
+        for s in state_match:
+            with suppress(ValueError):
+                completion_times.append(float(s.group('end')))
+
+        if completion_times:
+            self._completion_time = max(completion_times)
+
         return self._completion_time
 
     def _format_option(self, var, option):
diff --git a/reframe/core/schedulers/torque.py b/reframe/core/schedulers/torque.py
@@ -35,6 +35,13 @@
 class TorqueJobScheduler(PbsJobScheduler):
     TASKS_OPT = '-l nodes={num_nodes}:ppn={num_cpus_per_node}'
 
+    def _set_nodelist(self, job, nodespec):
+        if job.nodelist is not None:
+            return
+
+        job.nodelist = [x.split('/')[0] for x in nodespec.split('+')]
+        job.nodelist.sort()
+
     def _update_state(self, job):
         '''Check the status of the job.'''
 
@@ -53,6 +60,13 @@ def _update_state(self, job):
         if completed.returncode != 0:
             raise JobError('qstat failed: %s' % completed.stderr, job.jobid)
 
+        nodelist_match = re.search(
+            r'exec_host = (?P<nodespec>\S+)', completed.stdout
+        )
+        if nodelist_match:
+            nodespec = nodelist_match.group('nodespec')
+            self._set_nodelist(job, nodespec)
+
         state_match = re.search(
             r'^\s*job_state = (?P<state>[A-Z])', completed.stdout, re.MULTILINE
         )
diff --git a/reframe/frontend/cli.py b/reframe/frontend/cli.py
@@ -258,11 +258,6 @@ def main():
         help='Set the maximum number of times a failed regression test '
              'may be retried (default: 0)'
     )
-    run_options.add_argument(
-        '--flex-alloc-tasks', action='store',
-        dest='flex_alloc_tasks', metavar='{all|idle|NUM}', default=None,
-        help='*deprecated*, please use --flex-alloc-nodes instead'
-    )
     run_options.add_argument(
         '--flex-alloc-nodes', action='store',
         dest='flex_alloc_nodes', metavar='{all|idle|NUM}', default=None,
@@ -598,13 +593,6 @@ def print_infoline(param, value):
                                 "Skipping..." % m)
                 printer.debug(str(e))
 
-        if options.flex_alloc_tasks:
-            printer.warning("`--flex-alloc-tasks' is deprecated and "
-                            "will be removed in the future; "
-                            "you should use --flex-alloc-nodes instead")
-            options.flex_alloc_nodes = (options.flex_alloc_nodes or
-                                        options.flex_alloc_tasks)
-
         options.flex_alloc_nodes = options.flex_alloc_nodes or 'idle'
         if options.account:
             printer.warning(f"`--account' is deprecated and "
diff --git a/reframe/frontend/executors/__init__.py b/reframe/frontend/executors/__init__.py
diff --git a/reframe/frontend/executors/policies.py b/reframe/frontend/executors/policies.py
diff --git a/unittests/test_schedulers.py b/unittests/test_schedulers.py
