Document final retry behaviour. (#571)

Document final retry behaviour

Author: hjoliver

src/tutorial/furthertopics/retries.rst

@@ -8,14 +8,25 @@ failure in submission or execution.
 Purpose
 -------
 
-Retries can be useful for tasks that may occasionally fail due to external
-events, and are routinely fixable when they do - an example would be a task
-that is dependent on a system that experiences temporary outages.
-
-If a task fails, the Cylc retry mechanism can resubmit it after a
-pre-determined delay. An environment variable, ``$CYLC_TASK_TRY_NUMBER``
-is incremented and passed into the task - this means you can write your
-task script so that it changes behaviour accordingly.
+Retries can be useful for tasks that occasionally fail for known, fixable
+reasons. Cylc can rerun a failing job multiple times, with user-defined delays
+between tries.
+
+Tasks that fail because of temporary hardware or network outages may succeed if
+simply resubmitted after a delay. Others might succeed if configured differently
+on the retry.
+
+A job environment variable ``$CYLC_TASK_TRY_NUMBER`` increments with each try,
+to allow try-dependent behaviour in the task script.
+
+.. note::
+
+   Tasks only enter the ``submit-failed`` state if job submission fails with no
+   retries left. Otherwise they return to the waiting state, to wait on the
+   next try.
+
+   Tasks only enter the ``failed`` state if job execution fails with no retries
+   left. Otherwise they return to the waiting state, to wait on the next try.
 
 
 Example

src/tutorial/runtime/runtime-configuration.rst

@@ -160,7 +160,7 @@ Jobs can fail for several reasons:
    For example, setting ``execution retry delays = PT10M``
    will cause the job to retry every 10 minutes on execution failure.
 
-   Use a multiplier to limit retries to a specific number:
+   Use a multiplier to limit the number of retries:
 
 .. code-block:: cylc
 
@@ -176,6 +176,16 @@ Jobs can fail for several reasons:
          #   then every 30 mins thereafter.
          submission retry delays = 2*PT10M, PT30M
 
+.. note::
+
+   Tasks only enter the ``submit-failed`` state if job submission fails with no
+   retries left. Otherwise they return to the waiting state, to wait on the
+   next try.
+
+   Tasks only enter the ``failed`` state if job execution fails with no retries
+   left. Otherwise they return to the waiting state, to wait on the next try.
+
+
 
 .. _tutorial.start_stop_restart:
 

src/user-guide/running-workflows/retrying-tasks.rst

@@ -11,6 +11,18 @@ state, with a new clock trigger to handle the configured retry delay.
    A task that is waiting on a retry will already have one or more failed jobs
    associated with it.
 
+
+.. note::
+
+   Tasks only enter the ``submit-failed`` state if job submission fails with no
+   retries left. Otherwise they return to the waiting state, to wait on the
+   next try.
+
+   Tasks only enter the ``failed`` state if job execution fails with no retries
+   left. Otherwise they return to the waiting state, to wait on the next try.
+
+
+
 Aborting a Retry Sequence
 -------------------------
 

src/user-guide/writing-workflows/runtime.rst

@@ -504,6 +504,18 @@ Tasks can have a list of :term:`ISO8601 durations <ISO8601 duration>` as retry
 intervals. If the job fails the task will return to the ``waiting`` state
 with a clock-trigger configured with the next retry delay.
 
+
+.. note::
+
+   Tasks only enter the ``submit-failed`` state if job submission fails with no
+   retries left. Otherwise they return to the waiting state, to wait on the
+   next try.
+
+   Tasks only enter the ``failed`` state if job execution fails with no retries
+   left. Otherwise they return to the waiting state, to wait on the next try.
+
+
+
 In the following example, tasks ``bad`` and ``flaky`` each have 3 retries
 configured, with a 10 second delay between. On the final try, ``bad`` fails
 again and goes to the ``failed`` state, while ``flaky`` succeeds and triggers