user guide: flesh out workflow completion / stall sectionj

Author: oliver-sanders

src/glossary.rst

@@ -1373,7 +1373,7 @@ Glossary
      Every :term:`task` has a set of standard :term:`outputs <task output>`
      that trigger :term:`task state` changes:
 
-      - ``:expired```
+      - ``:expired``
       - ``:submitted``, or ``:submit-failed``
       - ``:started``
       - ``:succeeded``, or ``:failed``
@@ -1601,6 +1601,10 @@ Glossary
       User intervention is required to fix a stall, e.g. by retriggering
       tasks after fixing the problems that caused them to fail.
 
+      .. seealso:
+
+         * :ref:`Cylc User Guide <scheduler stall>`
+
 
    suicide trigger
       Suicide triggers remove tasks from the :term:`n=0 window <n-window>`.

src/img/gui-stall.png

@@ -63,6 +63,8 @@ Technical details:
   removed if necessary to allow the tasks to re-run in order.
 
 
+.. _changes.warning_triangles:
+
 Warning Triangles
 ^^^^^^^^^^^^^^^^^
 

src/reference/changes.rst

@@ -3,28 +3,92 @@
 Workflow Completion
 ===================
 
-A workflow can :term:`shut down <shutdown>` once all
-:term:`active tasks <active task>` complete without spawning further
-downstream activity - i.e., when :term:`n=0 window <n-window>` empties out.
+Once Cylc has run all of the tasks in the :term:`graph` (i.e. once it has
+reached the end of the workflow and there are no tasks left), the workflow
+will shut down automatically.
+
+A workflow with no tasks left is said to have "completed".
+
+When you restart a workflow, it will restart in the same state it shut down in.
+So if you restart a completed workflow (one with no remaining tasks), it will
+come back with no tasks. Having no more tasks to run, the workflow will
+automatically shut down after the configured
+:cylc:conf:`restart timeout <[scheduler][events]restart timeout>`.
+
+If you want to re-run some tasks in a completed workflow, restart the workflow
+then
+:ref:`re-trigger the selected tasks <interventions.re-run-multiple-tasks>`
+or :ref:`trigger a new flow <interventions.reflow>` to run through the graph
+(before the restart timeout passes).
+
+A common pattern is to restart a completed workflow and extend it for a few
+cycles. The easiest way to achieve this is to use the
+:cylc:conf:`stop after cycle point <[scheduling]stop after cycle point>`
+rather than the
+:cylc:conf:`final cycle point <[scheduling]final cycle point>`, this prevents
+the workflow from completing, making it easier to restart it from where it
+left off. For a worked example, see :ref:`examples.extending-workflow`.
+
 
 .. _scheduler stall:
 
 Scheduler Stall
 ===============
 
-A workflow has stalled if:
+If Cylc is unable to make progress through the :term:`graph` (i.e, if the path
+through the graph is "blocked"), then the workflow is considered
+:term:`stalled <stall>`.
 
-* No tasks are waiting on unsatisfied external events, like clock triggers and xtriggers.
-* AND All activity has ceased.
-* AND The workflow has not run to completion.
+Stalls are usually caused by unexpected task failures.
 
-A workflow which has stalled requires manual intervention to continue.
+A stalled workflow has not run to completion but cannot continue without manual
+intervention. Typically this involves
+:ref:`fixing and rerunning a failed task <interventions.edit-a-tasks-configuration>`.
+
+
+Stall Conditions
+----------------
+
+A workflow has stalled if:
+
+* The workflow has not run to completion (i.e, there are still tasks left
+  for Cylc to run).
+* AND no tasks are waiting on unsatisfied
+  :ref:`external events <Section External Triggers>` (e.g, clock triggers
+  and xtriggers).
+* AND All activity has ceased (i.e, no preparing, submitted or running tasks).
 
 Stalls are caused by :term:`final status incomplete tasks <output completion>`
 and :term:`partially satisfied tasks <prerequisite>`.
 
 These most often result from task failures that the workflow does not
-handle automatically by retries or optional branching.
+handle automatically by :term:`retries <retry>` or :term:`graph branching`.
+
+
+Diagnosing Stalls
+-----------------
+
+A screenshot of the Cylc GUI displaying a stalled workflow:
+
+.. image:: ../../img/gui-stall.png
+   :align: center
+   :width: 90%
+
+|
+
+In the above screenshot:
+
+* The stall was caused by the failure of the task ``2/a``.
+* The stall event is recorded in the :term:`workflow log` file (shown on the
+  right) along with the list of :term:`incomplete tasks <output completion>`
+  that caused it (2/a did not complete the required outputs: succeeded).
+* In the GUI, the :ref:`warning triangle <changes.warning_triangles>`
+  will light up to notify you of the error, hover over it to see the log
+  messages.
+
+
+Stall Timeouts
+--------------
 
 A stalled scheduler stays alive for a configurable timeout period
 to allow you to intervene, e.g. by manually triggering an incomplete
@@ -34,12 +98,28 @@ If a stalled workflow does eventually shut down, on the stall timeout
 or by stop command, it will immediately stall again on restart to await
 manual intervention.
 
-.. warning::
+Stall timeout behaviour is controlled by the following configurations:
+
+.. admonition:: Configuration
+   :class: note
+
+   :cylc:conf:`[scheduler][events]stall timeout`
+      The length of time before a stalled workflow will shut down.
+   :cylc:conf:`[scheduler][events]abort on stall timeout`
+      Whether the scheduler should shut down immediately with error status if
+      the stall timeout is reached.
+
+
+Stall Events
+------------
 
-   Look in the :term:`scheduler log` to see which tasks caused a stall.
+Cylc emits the :ref:`stall <user_guide.workflow_events.stall>` event when a
+scheduler stalls.
 
-.. seealso::
+.. admonition:: Configuration
+   :class: note
 
-   * :cylc:conf:`[scheduler][events]stall timeout`
-   * :cylc:conf:`[scheduler][events]abort on stall timeout`
-   * :cylc:conf:`[scheduler][events]stall handlers`
+   :cylc:conf:`[scheduler][events]mail events = stall`
+      Configure emails for stall events.
+   :cylc:conf:`[scheduler][events]stall handlers`
+      Configure custom event handlers to run on stall events.

src/user-guide/running-workflows/workflow-completion.rst

@@ -92,6 +92,8 @@ Some workflow events have related configurations e.g. for setting the timeout.
 
 .. describe:: stall
 
+   .. _user_guide.workflow_events.stall:
+
    :Event Handler: `stall handlers`
 
    The workflow :term:`stalled <stall>` (i.e. the scheduler cannot make any