Consolidate task polling documentation and correct the info on regular polling (#556)

Co-authored-by: dpmatthews <[email protected]>

Author: oliver-sanders

src/user-guide/running-workflows/tracking-task-state.rst

@@ -1,40 +1,3 @@
-
-.. _Task Job Polling:
-
-Job Polling
------------
-
-At any point after job submission, jobs can be *polled* to check that
-their true state matches what scheduler expects based on received job status
-messages or previous polls.
-
-Polling may be necessary if, for example, a job gets killed by the
-untrappable SIGKILL signal (e.g. ``kill -9 PID``), or after a network
-outage that prevents job status messages getting back to the scheduler, or if
-the :term:`scheduler` itself was down when active jobs finished.
-
-To poll a job the :term:`scheduler` interrogates the :term:`job runner`, and
-the ``job.status`` file of the task, on the job host. This information is
-enough to determine correct task status even if the job finished while the
-:term:`scheduler` was down or unreachable on the network.
-
-.. seealso::
-   - ``cylc poll --help``
-
-
-Routine Polling
-^^^^^^^^^^^^^^^
-
-Jobs are automatically polled at certain times: once on job submission
-timeout; several times on exceeding the job execution time limit; and at
-workflow restart any tasks recorded as active are polled to find out what
-happened to them while the workflow was down.
-
-Routine polling can also be configured as a way to track job status on platforms
-that do not allow routing back to the workflow host for task messaging by TCP
-or SSH. See :ref:`Polling To Track Job Status`.
-
-
 .. _TaskComms:
 
 Tracking Job Status
@@ -157,16 +120,3 @@ the last value, which is used repeatedly until the job is finished:
    [runtime]
       [[task]]
          platform = my_platform
-
-
-A list of intervals with optional multipliers can be used for both submission
-and execution polling, although a single value is probably sufficient for
-submission. If these items are not configured, default values from
-site and user global config will be used for
-:cylc:conf:
-`global.cylc[platforms][<platform name>]communication method = poll`.
-
-Polling is not done by default under the other task communications methods, but
-it can be configured as well if you like.
-
-

src/user-guide/task-implementation/job-submission.rst

@@ -174,19 +174,27 @@ The template's ``%(job)s`` will be substituted by the job file path.
 Job Polling
 -----------
 
-For supported :term:`job runners <job runner>`, one-way polling can be used to determine actual
-job status: the :term:`scheduler` executes a process on the task host, by
-non-interactive ssh, to interrogate the batch queueing system there, and to
-read a *status file* that is automatically generated by the :term:`job script`
-as it runs.
+For supported :term:`job runners <job runner>`, jobs can be *polled* to
+check that their true state matches what the scheduler expects based on received
+job status messages or previous polls. The :term:`scheduler` executes a process
+on the task host, by non-interactive ssh, to interrogate the job runner, and to
+read the ``job.status`` file of the task which is automatically generated by the
+:term:`job script` as it runs.
 
 Polling may be required to update the workflow state correctly after unusual
-events such as a machine being rebooted with tasks running on it, or network
-problems that prevent task messages from getting back to the workflow host.
+events such as
+
+- a job gets killed by the untrappable SIGKILL signal (e.g. ``kill -9 PID``)
+- a machine being rebooted with tasks running on it
+- network problems prevent task messages from getting back to the workflow host
+- the :term:`scheduler` itself was down when active jobs finished
 
 Tasks can be polled on demand by using the
 ``cylc poll`` command.
 
+.. seealso::
+   - ``cylc poll --help``
+
 Tasks are polled automatically, once, if they timeout while queueing in a
 job runner and submission timeout is set.
 (See :cylc:conf:`[runtime][<namespace>][events]`
@@ -196,34 +204,28 @@ Tasks are polled multiple times, where necessary, when they exceed their
 execution time limits. These are normally set with some initial delays to allow
 the job runners to kill the jobs.
 (See
-:cylc:conf:`execution time limit intervals <global.cylc[platforms][<platform name>]execution time limit polling intervals>`
+:cylc:conf:`execution time limit polling intervals <global.cylc[platforms][<platform name>]execution time limit polling intervals>`
 for how to configure the polling intervals).
 
 Any tasks recorded in the *submitted* or *running* states at workflow
 restart are automatically polled to determine what happened to them while the
 workflow was down.
 
-Regular polling can also be configured as a health check on tasks submitted to
-hosts that are known to be flaky, or as the sole method of determining task
-status on hosts that do not allow task messages to be routed back to the workflow
-host.
-
-To use polling instead of task-to-workflow messaging set
-:cylc:conf:`global.cylc[platforms][<platform name>]communication method = poll`.
-
-The default polling intervals can be overridden in the global configuration:
+By default, regular polling also takes place every 15 minutes whilst a job is
+submitted or running. The default polling intervals can be overridden in the
+global configuration:
 
-* :cylc:conf:`submission polling intervals<global.cylc[platforms][<platform name>]submission polling intervals>`
-* :cylc:conf:`execution polling intervals<global.cylc[platforms][<platform name>]execution polling intervals>`
+* :cylc:conf:`global.cylc[platforms][<platform name>]submission polling intervals`
+* :cylc:conf:`global.cylc[platforms][<platform name>]execution polling intervals`
 
-Or in workflow configurations (in which case polling will be done regardless
-of the communication method configured for the platform):
+The polling intervals can also be configured for individual tasks:
 
-* :cylc:conf:`submission polling intervals<[runtime][<namespace>]submission polling intervals>`
-* :cylc:conf:`execution polling intervals<[runtime][<namespace>]execution polling intervals>`
+* :cylc:conf:`[runtime][<namespace>]submission polling intervals`
+* :cylc:conf:`[runtime][<namespace>]execution polling intervals`
 
-Note that regular polling is not as efficient as task messaging in updating
-task status, and it should be used sparingly in large workflows.
+Polling can be used as the sole method of determining task status on hosts that
+do not allow task messages to be routed back to the workflow host.
+See :ref:`Polling To Track Job Status`.
 
 .. note::