Skip to content

Commit 5b5263c

Browse files
oliver-sandersoliver-sanders
committed
examples: document expiry workflow design patterns
1 parent 1593ef3 commit 5b5263c

File tree

5 files changed

+310
-0
lines changed

5 files changed

+310
-0
lines changed

‎cylc/flow/etc/examples/expiry/.validate‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
#!/bin/bash
2+
# THIS FILE IS PART OF THE CYLC WORKFLOW ENGINE.
3+
# Copyright (C) NIWA & British Crown (Met Office) & Contributors.
4+
#
5+
# This program is free software: you can redistribute it and/or modify
6+
# it under the terms of the GNU General Public License as published by
7+
# the Free Software Foundation, either version 3 of the License, or
8+
# (at your option) any later version.
9+
#
10+
# This program is distributed in the hope that it will be useful,
11+
# but WITHOUT ANY WARRANTY; without even the implied warranty of
12+
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
13+
# GNU General Public License for more details.
14+
#
15+
# You should have received a copy of the GNU General Public License
16+
# along with this program. If not, see <http://www.gnu.org/licenses/>.
17+
18+
set -euxo pipefail
19+
20+
test_one () {
21+
ID="$(< /dev/urandom tr -dc A-Za-z | head -c6 || true)"
22+
23+
# start the workflow
24+
cylc vip \
25+
--check-circular \
26+
--no-detach \
27+
--final-cycle-point=P0D \
28+
--no-run-name \
29+
--workflow-name "$ID" \
30+
./one
31+
32+
# the start task should have expired
33+
grep 'start.*(internal)expired' "$HOME/cylc-run/$ID/log/scheduler/log"
34+
35+
# the following task(s) should not have run
36+
grep 'a.*running' "$HOME/cylc-run/$ID/log/scheduler/log" && exit 1
37+
grep 'b.*running' "$HOME/cylc-run/$ID/log/scheduler/log" && exit 1
38+
39+
# lint
40+
cylc lint "$ID"
41+
42+
# clean up
43+
cylc clean "$ID"
44+
}
45+
46+
47+
test_two () {
48+
ID="$(< /dev/urandom tr -dc A-Za-z | head -c6 || true)"
49+
50+
# start the workflow
51+
cylc vip \
52+
--check-circular \
53+
--no-detach \
54+
--final-cycle-point=P0D \
55+
--no-run-name \
56+
--workflow-name "$ID" \
57+
./two
58+
59+
# the start task should run
60+
grep 'start.*running' "$HOME/cylc-run/$ID/log/scheduler/log"
61+
62+
# some other task in the chain should expire
63+
grep '(internal)expired' "$HOME/cylc-run/$ID/log/scheduler/log"
64+
65+
# the housekeep task at the end of the cycle should not run
66+
grep 'housekeep.*running' "$HOME/cylc-run/$ID/log/scheduler/log" && exit 1
67+
68+
# lint
69+
cylc lint "$ID"
70+
71+
# clean up
72+
cylc clean "$ID"
73+
}
74+
75+
76+
test_three () {
77+
ID="$(< /dev/urandom tr -dc A-Za-z | head -c6 || true)"
78+
79+
# start the workflow
80+
cylc vip \
81+
--check-circular \
82+
--no-detach \
83+
--final-cycle-point=P0D \
84+
--no-run-name \
85+
--workflow-name "$ID" \
86+
./three
87+
88+
# the start task should expire
89+
grep 'start.*(internal)expired' "$HOME/cylc-run/$ID/log/scheduler/log"
90+
# shellcheck disable=SC2125 # could only ever be one matching file
91+
local job_file="$HOME/cylc-run/$ID/log/job/"*"/a/NN/job"
92+
[[ ! -f "$job_file" ]]
93+
94+
# only the "a" and "housekeep" tasks should run
95+
[[ $(cd "$HOME/cylc-run/$ID/log/job/"*; echo *) == 'a housekeep' ]]
96+
97+
# tasks b, c and d should skip
98+
grep '\/b.*run mode=skip' "$HOME/cylc-run/$ID/log/scheduler/log"
99+
grep '\/c.*run mode=skip' "$HOME/cylc-run/$ID/log/scheduler/log"
100+
grep '\/d.*run mode=skip' "$HOME/cylc-run/$ID/log/scheduler/log"
101+
102+
# lint
103+
cylc lint "$ID"
104+
105+
# clean up
106+
cylc clean "$ID"
107+
}
108+
109+
110+
test_one
111+
test_two
112+
test_three

‎cylc/flow/etc/examples/expiry/index.rst‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
.. _examples.expiry:
2+
3+
Expiring Tasks / Cycles
4+
-----------------------
5+
6+
Cylc is often used to write workflows which monitor real-world events.
7+
8+
For example, this workflow will run the task ``foo`` every day at 00:00am:
9+
10+
.. code-block:: cylc
11+
12+
[scheduling]
13+
initial cycle point = previous(T00)
14+
[[graph]]
15+
P1D = """
16+
@wall_clock => foo
17+
"""
18+
19+
Sometimes such workflows might get behind, e.g. due to failures or slow task
20+
execution. In this situation, it might be necessary to skip a few tasks in
21+
order for the workflow to catch up with the real-world time.
22+
23+
Cylc has a concept called :ref:`expiry <ClockExpireTasks>` which allows tasks
24+
to be automatcially "expired" if they are running behind schedule. The expiry
25+
can be configred as an offset from the cycle time.
26+
27+
.. seealso::
28+
29+
:ref:`ClockExpireTasks`.
30+
31+
32+
Example 1: Skip a whole cycle of tasks
33+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
34+
35+
If the workflow gets behind, skip whole cycles of tasks until it catches up.
36+
37+
.. admonition:: Get a copy of this example
38+
:class: hint
39+
40+
.. code-block:: console
41+
42+
$ cylc get-resources examples/expiry/one
43+
44+
.. literalinclude:: one/flow.cylc
45+
:language: cylc
46+
47+
48+
Example 2: Skip the remainder of a cycle of tasks
49+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
50+
51+
If the workflow gets behind, skip the remainder of the tasks in the cycle,
52+
then skip whole cycles of tasks until it catches up.
53+
54+
.. admonition:: Get a copy of this example
55+
:class: hint
56+
57+
.. code-block:: console
58+
59+
$ cylc get-resources examples/expiry/two
60+
61+
.. literalinclude:: two/flow.cylc
62+
:language: cylc
63+
64+
65+
Example 3: Skip selected tasks in a cycle
66+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
67+
68+
If the workflow gets behind, turn off selected tasks to allow it to catch up
69+
more quickly.
70+
71+
.. admonition:: Get a copy of this example
72+
:class: hint
73+
74+
.. code-block:: console
75+
76+
$ cylc get-resources examples/expiry/three
77+
78+
.. literalinclude:: three/flow.cylc
79+
:language: cylc

‎cylc/flow/etc/examples/expiry/one/flow.cylc‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
[meta]
2+
description = """
3+
If the workflow runs slowly and the cycle time gets behind the real
4+
world (wallclock) time, then it will skip cycles until it catches up.
5+
6+
Either a cycle runs or it is skipped.
7+
8+
When you start this workflow, the first cycle will be at 00:00am this
9+
morning so will immediately expire causing the workflow to move onto
10+
tomorrow's cycle.
11+
"""
12+
13+
[scheduler]
14+
allow implicit tasks = True
15+
16+
[scheduling]
17+
# start the workflow at 00:00am this morning
18+
initial cycle point = previous(T00)
19+
20+
# the "start" task will "expire" if the cycle time falls behind
21+
# the wallclock time
22+
[[special tasks]]
23+
clock-expire = start
24+
25+
[[graph]]
26+
P1D = """
27+
# the chain of tasks we want to run
28+
start => a => b => c => d => housekeep
29+
30+
# wait for the previous cycle to either complete or expire before
31+
# continuing onto the next cycle
32+
housekeep[-P1D] | start[-P1D]:expired? => start
33+
"""

‎cylc/flow/etc/examples/expiry/three/flow.cylc‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
[meta]
2+
description = """
3+
If the workflow runs slowly and the cycle time gets behind the real
4+
world (wallclock) time, then it will skip selected tasks until it
5+
catches up.
6+
7+
In this case, the tasks "b", "c" and "d" will be skipped to help the
8+
workflow to catch up more quickly.
9+
10+
When this workflow starts up, the first cycle will be at 00:00am today
11+
so the "start" task will immediately expire. This will cause tasks
12+
"b", "c" and "d" to be configured to "skip" rather than run.
13+
"""
14+
15+
[scheduler]
16+
allow implicit tasks = True
17+
18+
[scheduling]
19+
# start the workflow at 00:00am this morning
20+
initial cycle point = previous(T00)
21+
final cycle point = +P0D
22+
23+
# the "start" task will "expire" if the cycle time falls behind
24+
# the wallclock time
25+
[[special tasks]]
26+
clock-expire = start
27+
28+
[[graph]]
29+
P1D = """
30+
# the chain of tasks we want to run
31+
start | start:expired? => a => b => c => d => housekeep
32+
"""
33+
34+
[runtime]
35+
[[start]]
36+
# if this task expires, configure the tasks "b", "c" and "d" to
37+
# "skip" rather than run
38+
# Note: This task will also be "skipped" if it expires
39+
[[[events]]]
40+
expired handlers = cylc broadcast "%(workflow)s" -p "%(point)s" -n b -n c -n d -s "run mode = skip"

‎cylc/flow/etc/examples/expiry/two/flow.cylc‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
[meta]
2+
description = """
3+
If the workflow runs slowly and the cycle time gets behind the real
4+
world (wallclock) time, then it will skip tasks until it catches up.
5+
6+
A cycle may be skipped part way through to allow the workflow to catch
7+
up faster.
8+
9+
When this workflow starts up, the first cycle will be one minute ahead
10+
of the wallclock time. At some point in the cycle, the wallclock time
11+
will overtake the cycle time and the next task in the chain will
12+
expire. The workflow will then move onto the next cycle.
13+
"""
14+
15+
[scheduler]
16+
allow implicit tasks = True
17+
18+
[scheduling]
19+
# start the workflow at 00:00am this morning
20+
initial cycle point = PT1M
21+
22+
# any task in the workflow will "expire" rather than run if the cycle
23+
# time falls behind the wallclock time
24+
[[special tasks]]
25+
clock-expire = start, a, b, c, d, housekeep
26+
27+
[[graph]]
28+
P1D = """
29+
# the chain of tasks we want to run
30+
start => a => b => c => d => housekeep
31+
32+
# start the next cycle as soon as the previous cycle has finished
33+
# OR and task in the previous cycle has expired
34+
housekeep[-P1D]
35+
| start[-P1D]:expire?
36+
| a[-P1D]:expired?
37+
| b[-P1D]:expired?
38+
| c[-P1D]:expired?
39+
| d[-P1D]:expired?
40+
| housekeep[-P1D]:expired?
41+
=> start
42+
"""
43+
44+
[runtime]
45+
[[root]]
46+
script = sleep 12

0 commit comments

Comments
 (0)