Merge pull request #528 from oliver-sanders/task-job

Task job

Author: oliver-sanders

src/7-to-8/major-changes/platforms.rst

@@ -6,46 +6,11 @@ Platforms
 .. admonition:: Does This Change Affect Me?
    :class: tip
 
-   .. cylc-scope:: flow.cylc
+   Cylc platforms are a new feature which replace the task ``job`` and
+   ``remote`` configuration sections:
 
-   Platforms replace the deprecated :cylc:conf:`[runtime][<namespace>][job]`
-   and :cylc:conf:`[runtime][<namespace>][remote]`
-   sections:
-
-   .. code-block:: cylc
-
-      [runtime]
-          [[foo]]
-              [[[job]]]
-                  batch system = slurm
-              [[[remote]]]
-                  host = my_supercomputer
-
-   .. cylc-scope::
-
-   Read this section if your workflow's jobs run on a remote computer or if
-   you see the following warning on running ``cylc validate``:
-
-   .. code-block:: console
-
-      WARNING - deprecated settings found (please replace with [runtime][foo]platform):
-          [runtime][foo][remote]host
-          [runtime][foo][job]batch system
-
-   If you currently use the ``rose host-select`` utility or a similar host
-   selection or load balancing utility the intelligent host selection
-   functionality of Cylc 8 may be used instead:
-
-   .. code-block:: cylc
-
-      [runtime]
-          [[task1]]
-              [[[remote]]]
-                  host = $(rose host-select my-computer)
-          [[task2]]
-              # An example of a home-rolled host selector
-              [[[remote]]]
-                  host = $(test $((RANDOM%2)) -eq 0 && echo "host_a" || echo "host_b")
+   * :cylc:conf:`[runtime][<namespace>][job]`
+   * :cylc:conf:`[runtime][<namespace>][remote]`
 
 
 Overview
@@ -57,23 +22,17 @@ Overview
    - The terms :term:`job runner` (in Cylc 8 configurations) and batch system
      (in Cylc 7 configurations) are equivalent.
 
-Cylc 7 defines settings for remote :term:`jobs <job>` in each
-:term:`task's <task>` definition.
-
-Cylc 8 allows site administrators (and users) to configure
-:term:`platforms <platform>` in ``global.cylc``. A platform can have
-multiple hosts with associated platform-specific settings. Users only need to
-select the platform for their task jobs.
+Submitting a job to a :term:`job runner` may require configuration.
 
-Platforms also define how hosts are selected from each platform:
+In Cylc 7 this configuration must be provided for each task in the workflow
+configuration (``suite.rc``).
 
-- Randomly (default)
-- By definition order
+In Cylc 8 "platforms" can be defined in the global configuration
+(:cylc:conf:`global.cylc`) so that this configuration doesn't have to be
+repeated for each task in each workflow.
 
-There may be cases where sets of platforms (for example a group of
-standalone compute servers, or a pair of mirrored HPC's) might be equally
-suitable for a task, but not share files systems to allow them to constitute
-a single platform. Such platforms can be set up to be ``platform groups``
+Cylc "platforms" may configure hostnames, job runners and more. Only the
+platform name needs to be specified in the task configuration.
 
 .. seealso::
 
@@ -92,6 +51,20 @@ a single platform. Such platforms can be set up to be ``platform groups``
    Deprecated settings will be removed in a later release of Cylc.
 
 
+What is a Platform?
+-------------------
+
+A "platform" represents one or more hosts from which jobs can be submitted to or
+polled from a common job submission system.
+
+If a platform has multiple hosts Cylc will automatically select a host when
+needed and will fallback to other hosts if it is not contactable.
+
+A "platform group" represents a collection of independent platforms. Cylc will
+automatically select a platform and will fallback to other platforms in the
+group (for appropriate operations) if the platform is not contactable.
+
+
 Examples
 --------
 

src/7-to-8/major-changes/task-job-states.rst

@@ -14,11 +14,11 @@ the submission stage, real job scripts). A task can have multiple jobs, by
 automatic retries and manual re-triggering.
 
 Cylc 7 had 13 task/job states. The GUI only showed tasks, with job data
-from the latest task job.
+from the latest job.
 
 Cylc 8 has only 8 task/job states. The Cylc 8 UI shows both task and jobs.
 Task icons are monochrome circles; job icons are coloured squares. The running
 task icon incorporates a radial progress indicator.
 
 .. image:: ../../img/task-job.png
-   :align: center
+   :align: center

src/7-to-8/summary.rst

@@ -97,7 +97,7 @@ Task/Job States
 
 :term:`Tasks <task>` are nodes in the abstract workflow graph, representing
 applications to run at the appropriate point in the workflow. A :term:`job <job>`
-(or a *task job*) is the script (and subsequent process) submitted by Cylc to
+is the script (and subsequent process) submitted by Cylc to
 actually run the application. A task can have multiple jobs as the result of
 automatic retries or manual re-triggering.
 
@@ -146,7 +146,7 @@ Cylc 8 is aware of sets of host settings called
 
 Hosts of a platform must share a file system and :term:`job runner`:
 If one host is unavailable Cylc 8 can use other hosts
-on the same platform to interact with task jobs.
+on the same platform to interact with jobs.
 
 The same hosts can belong to multiple platforms, for example
 you might be able to use the same host to launch both background and Slurm
@@ -250,7 +250,7 @@ Queues
 Time Zones
    :cylc:conf:`[scheduler]cycle point time zone` now defaults to UTC, unless you
    are working in :ref:`cylc_7_compat_mode`.
-Task Job Scripts
+Job Scripts
    All user-defined task scripting now runs in a subshell, so you can safely
    switch Python environments inside tasks without affecting Cylc.
    Further information is available in the User Guide: :ref:`JobScripts`.

src/glossary.rst

@@ -136,7 +136,7 @@ Glossary
       The workflow name is a path relative to the cylc-run directory which
       contains one or more workflow :term:`run directories <run directory>`.
 
-      Task jobs can get the workflow name from ``$CYLC_WORKFLOW_NAME`` in their
+      Tasks can get the workflow name from ``$CYLC_WORKFLOW_NAME`` in their
       runtime environment.
 
       Unlike :term:`workflow id` the name is not always a unique identifier. In
@@ -712,7 +712,7 @@ Glossary
 
          <run-directory>/work/<cycle-point>/<task-name>
 
-      Task jobs can get their own work directory path at runtime from
+      Tasks can get their own work directory path at runtime from
       the ``CYLC_TASK_WORK_DIR`` environment variable or the Posix ``pwd``
       command.
 
@@ -731,7 +731,7 @@ Glossary
 
          <run-directory>/share
 
-      Task jobs can get their own share directory path at runtime from
+      Tasks can get their own share directory path at runtime from
       the ``CYLC_WORKFLOW_SHARE_DIR`` environment variable.
 
       In cycling workflows files are typically stored in cycle point
@@ -784,7 +784,7 @@ Glossary
          <run-directory>/log/job/<cycle-point>/<task-name>/<job-submit-num>
 
 
-      You can print task job logs at the terminal with ``cylc cat-log
+      You can print job logs at the terminal with ``cylc cat-log
       <workflow-name> <task-id>``. By default this prints ``job.out``.
       There are command options to select the other logs.
 

src/installation.rst

@@ -120,7 +120,7 @@ your ``$PATH``, follow the instructions in the ``brew install`` output.
 
    Newer version of Mac OS set ``zsh`` as the default shell (as opposed to
    ``bash``). You do not need to change this but be aware that Cylc uses
-   ``bash`` (for task job scripts) which has a subtly different syntax.
+   ``bash`` (for job scripts) which has a subtly different syntax.
 
 .. warning::
 
@@ -166,7 +166,7 @@ User Machines
 Cylc Servers
    Where Cylc schedulers run to manage workflows.
 Job Hosts
-   Where task jobs run, e.g. supercomputers or clusters
+   Where jobs run, e.g. supercomputers or clusters
 
 .. note::
 
@@ -258,9 +258,9 @@ configuration of the system on both a site and user basis.
 Bash Profile
 ^^^^^^^^^^^^
 
-Cylc task job scripts are bash scripts, which is good for manipulating files
-and processes, They invoke ``bash -l`` to allow environment configuration in
-login scripts.
+Cylc :term:`job scripts <job script>` are bash scripts, which is good for
+manipulating files and processes, They invoke ``bash -l`` to allow environment
+configuration in login scripts.
 
 .. warning::
 

src/reference/config/writing-platform-configs.rst

@@ -40,9 +40,8 @@ What Are Platforms?
 Platforms define settings, most importantly:
 
  - A set of ``hosts``.
- - A ``job runner`` (formerly a ``batch system``) where Cylc can submit a
-   task job.
- - An ``install target`` for Cylc to install task job files on.
+ - A ``job runner`` (formerly a ``batch system``) where Cylc can submit a job.
+ - An ``install target`` for Cylc to install job files on.
 
 Why Were Platforms Introduced?
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

src/reference/dev-history-major-changes.rst

@@ -30,7 +30,7 @@ Cylc Development History - Major Changes
   - multi-threading for continuous network request handling and job submission
   - more task states to distinguish job submission from execution
   - dependence between suites via new run databases
-  - polling and killing of real task jobs
+  - polling and killing of real jobs
   - polling as task communications option
 
 - **cylc-6**

src/reference/job-script-vars/var-list.txt

@@ -29,7 +29,7 @@ CYLC_WORKFLOW_SHARE_DIR            # Workflow (or task!) shared directory (see b
 CYLC_WORKFLOW_UUID                 # Workflow UUID string
 CYLC_WORKFLOW_WORK_DIR             # Workflow work directory (see below)
 
-CYLC_TASK_JOB                      # Task job identifier expressed as
+CYLC_TASK_JOB                      # Job identifier expressed as
                                    # CYCLE-POINT/TASK-NAME/SUBMIT-NUMBER
                                    #   e.g. 20110511T1800Z/t1/01
 
@@ -50,7 +50,7 @@ CYLC_TASK_FLOW_NUMBERS             # Flows this task belongs to, e.g. 1,2
 
 CYLC_TASK_LOG_DIR                  # Location of the job log directory
                                    #   e.g. ~/cylc-run/foo/log/job/20110511T1800Z/t1/01/
-CYLC_TASK_LOG_ROOT                 # The task job file path
+CYLC_TASK_LOG_ROOT                 # The job script path
                                    #   e.g. ~/cylc-run/foo/log/job/20110511T1800Z/t1/01/job
 CYLC_TASK_WORK_DIR                 # Location of task work directory (see below)
                                    #   e.g. ~/cylc-run/foo/work/20110511T1800Z/t1

src/tutorial/runtime/introduction.rst

@@ -85,7 +85,7 @@ We can also call other scripts or executables in this way, e.g:
 
    To help with this, Cylc automatically adds a ``bin/`` sub-directory of the
    workflow :term:`source directory` to the executable search path (``$PATH``)
-   in task job environments.
+   in task environments.
 
 .. code-block:: bash
    :caption: bin/hello_world
@@ -131,7 +131,7 @@ Tasks And Jobs
    When a :term:`task` is ready to run Cylc creates a :term:`job script` for
    it: a bash file containing the scripting defined for the task along with
    other configuration and error trapping code. This is what actually executes
-   as the task job.
+   as the job.
 
    :term:`Tasks<task>` typically go through the following states as a workflow
    runs:
@@ -141,11 +141,11 @@ Tasks And Jobs
    Preparing
       Dependencies met; preparing the task :term:`job script` for submission.
    Submitted
-      Task job script submitted to the :term:`job runner`; waiting on execution.
+      Job script submitted to the :term:`job runner`; waiting on execution.
    Running
-      Task job script executing.
+      Job script executing.
    Succeeded
-      Task job completed successfully (i.e. exited with 0 return status).
+      Job completed successfully (i.e. exited with 0 return status).
 
    There are several other task states as well, such as **failed**.
 
@@ -418,14 +418,14 @@ Files Generated at Runtime
       read or write files shared with other tasks.
    ``work/``
       Contains task :term:`work directories <work directory>`, i.e. the
-      *current working directories* of running task jobs. These are
+      *current working directories* of running tasks. These are
       removed automatically if empty when a task finishes.
 
     The job log directory path ends in ``<cycle-point>/<task-name>/<job-submission-num>/``,
     where the :term:`job submission number` starts at 1 and increments each time a
     task re-runs.
 
-    You can use the command line to view scheduler or task job logs without
+    You can use the command line to view scheduler or job logs without
     having to find them yourself on the filesystem:
 
     .. code-block:: bash
@@ -540,7 +540,7 @@ Files Generated at Runtime
       .. code-block:: none
 
          Workflow    : runtime-introduction
-         Task Job : 20000101T0000Z/get_observations_heathrow/01 (try 1)
+         Job : 20000101T0000Z/get_observations_heathrow/01 (try 1)
          User@Host: username@hostname
 
          Guessing Weather Conditions

src/tutorial/runtime/runtime-configuration.rst

@@ -56,7 +56,7 @@ Job Submission
 
 .. ifnotslides::
 
-   By default Cylc runs :term:`task jobs <job>` on the same machine as
+   By default Cylc runs :term:`jobs <job>` on the same machine as
    the scheduler. It can run them on other machines too if we set the
    :term:`platform` like this:
 
@@ -117,7 +117,7 @@ Time Limits
 .. ifnotslides::
 
    We can specify an execution time limit, as an :term:`ISO8601 duration`, after
-   which a task job will be terminated. Cylc automatically translates this to
+   which a job will be terminated. Cylc automatically translates this to
    the correct :term:`job runner` directives.
 
 .. code-block:: cylc
@@ -131,7 +131,7 @@ Time Limits
 Retries
 -------
 
-Task jobs can fail for several reasons:
+Jobs can fail for several reasons:
 
 .. nextslide::