Merge pull request #395 from oliver-sanders/workflow-design-guide-8.0.0

design guide: update for cylc 8

Author: hjoliver

src/workflow-design-guide/general-principles.rst

@@ -50,38 +50,23 @@ Monolithic Or Interdependent Workflows
 When writing workflows from scratch you may need to decide between putting
 multiple loosely connected sub-workflows into a single large workflow, or
 constructing a more modular system of smaller workflows that depend on each other
-through inter-workflow triggering. Each approach has its pros and cons, depending
-on your requirements and preferences with respect to the complexity and
-manageability of the resulting system.
-
-Inter-Workflow Triggering
-^^^^^^^^^^^^^^^^^^^^^^^^^
-
-A task in one workflow can explicitly trigger off of a task in another workflow. The
-full range of possible triggering conditions is supported, including custom
-message triggers. Remote triggering involves repeatedly querying ("polling")
-the remote workflow run database, not the :term:`scheduler`, so it works even
-if the other workflow is down at the time.
-
-There is special graph syntax to support triggering off of a task in another
-workflow, or you can call the underlying ``cylc workflow-state`` command
-directly in task scripting.
-
-In real time workflows you may want to use clock-triggers to delay the onset of
-inter-workflow polling until roughly the expected completion time of the remote
-task.
+through :ref:`inter-workflow triggering <Built-in Workflow State Triggers>`.
+Each approach has its pros and cons, depending on your requirements and
+preferences with respect to the complexity and manageability of the resulting
+system.
 
 
 .. _Self-Contained Workflows:
 
 Self-Contained Workflows
 ------------------------
 
-All files generated by Cylc during a workflow run are confined to the *workflow
-run directory* ``$HOME/cylc-run/<workflow-name>``. However, Cylc has no control
-over the locations of the programs, scripts, and files, that are executed,
-read, or generated by your tasks at runtime. It is up to you to ensure that
-all of this is confined to the run directory too, as far as possible.
+All files generated by Cylc during a workflow run are confined to the workflow
+:term:`run directory` ``$HOME/cylc-run/<workflow-id>``. However, Cylc has no
+control over the locations of the programs, scripts, and files, that are
+executed, read, or generated by your tasks at runtime. It is up to you to
+ensure that all of this is confined to the run directory too, as far as
+possible.
 
 Self-contained workflows are more robust, easier to work with, and more portable.
 Multiple instances of the same workflow (with different workflow names) should be
@@ -148,19 +133,16 @@ to copy or move selected files to external locations as needed (see
 Task Host Selection
 -------------------
 
-At sites with multiple task hosts to choose from, use
-``rose host-select`` to dynamically select appropriate task hosts
-rather than hard coding particular hostnames. This enables your workflow to
-adapt to particular machines being down or heavily overloaded by selecting
-from a group of hosts based on a series of criteria.
-``rose host-select`` will only return hosts that can be contacted by
-non-interactive SSH.
+The ``rose host-select`` command is now deprecated. Workflows should migrate
+to using :term:`platforms <platform>` which provide a more efficient
+solution.
+See :ref:`MajorChangesPlatforms` for details.
 
 
 Task Scripting
 --------------
 
-Non-trivial task scripting should be held in external files rather than
+Non-trivial task scripting should be held in separate script files rather than
 inlined in :cylc:conf:`flow.cylc`. This keeps the workflow definition tidy, and it
 allows proper shell-mode text editing and independent testing of task scripts.
 
@@ -274,29 +256,34 @@ development and testing.
 Clock-Triggered Tasks
 ---------------------
 
-Tasks that wait on real time data should use clock-triggers to delay job
-submission until the expected data arrival time:
+Tasks that wait on real time data should use
+:ref:`clock triggers <Built-in Clock Triggers>`
+to delay job submission until the expected data arrival time:
 
 .. code-block:: cylc
 
    [scheduling]
        initial cycle point = now
-       [[special tasks]]
+       [[xtriggers]]
            # Trigger 5 min after wallclock time is equal to cycle point.
-           clock-trigger = get-data(PT5M)
+           clock = wall_clock(offset=PT5M)
        [[graph]]
-           T00 = get-data => process-data
+           T00 = @clock => get-data => process-data
+
+.. cylc-scope:: flow.cylc[runtime][<namespace>]
 
 Clock-triggered tasks typically have to handle late data arrival. Task
-execution *retry delays* can be used to simply retrigger the task at
-intervals until the data is found, but frequently retrying small tasks probably
-should not go to a :term:`job runner`, and multiple task failures will be logged
-for what is a essentially a normal condition (at least it is normal until the
-data is really late).
+:cylc:conf:`execution retry delays` can be used to simply retrigger
+the task at intervals until the data is found, but frequently retrying small
+tasks is inefficient, and multiple task
+failures will be logged for what is a essentially a normal condition (at least
+it is normal until the data is really late).
+
+.. cylc-scope::
 
 Rather than using task execution retry delays to repeatedly trigger a task that
 checks for a file, it may be better to have the task itself repeatedly poll for
-the data (see :ref:`Rose App File Polling` for example).
+the data (see :ref:`Custom Trigger Functions`).
 
 
 .. _Rose App File Polling:
@@ -318,7 +305,8 @@ Task Execution Time Limits
 --------------------------
 
 Instead of setting job wallclock limits directly in :term:`job runner`
-directives, use the ``execution time limit`` workflow config item.
+directives, use
+:cylc:conf:`flow.cylc[runtime][<namespace>]execution time limit`.
 Cylc automatically derives the correct job runner directives from this,
 and it is also used to run ``background`` and ``at`` jobs via
 the ``timeout`` command, and to poll tasks that haven't reported in

src/workflow-design-guide/portable-workflows.rst

@@ -3,8 +3,6 @@
 Portable Workflows
 ==================
 
-.. TODO - platformise all the examples in here
-
 A *portable* or *interoperable* workflow can run "out of the box" at
 different sites, or in different environments such as research and operations
 within a site. For convenience we just use the term *site portability*.

src/workflow-design-guide/style-guide.rst

@@ -161,23 +161,8 @@ script:
 Graph String Lines
 ^^^^^^^^^^^^^^^^^^
 
-Multiline ``graph`` strings can be entirely free-form:
-
-.. code-block:: cylc
-
-   [scheduling]
-       [[graph]]
-           R1 = """
-       # Main workflow:
-     FAMILY:succeed-all => bar & baz => qux
-
-       # Housekeeping:
-     qux => rose_arch => rose_prune"""
-
-Whitespace is ignored in graph string parsing, however, so internal graph lines
-can be indented as if part of the :cylc:conf:`flow.cylc` syntax, or even out to the triple
-quotes, if you feel it aids readability (but watch line length with large
-indents; see :ref:`Line Length`):
+Whitespace is ignored in graph string parsing so internal graph lines
+should be indented as if part of the :cylc:conf:`flow.cylc` syntax:
 
 .. code-block:: cylc
 
@@ -191,8 +176,6 @@ indents; see :ref:`Line Length`):
                qux => rose_arch => rose_prune
            """
 
-Both styles are acceptable; choose one and use it consistently.
-
 
 Jinja2 Code
 ^^^^^^^^^^^
@@ -237,6 +220,8 @@ Titles, Descriptions, And URLs
 
 Document the workflow and its tasks with ``title``,
 ``description``, and ``url`` items instead of comments.
+See the :cylc:conf:`flow.cylc[meta]` and
+:cylc:conf:`flow.cylc[runtime][<namespace>][meta]` sections.
 
 
 .. _Line Length:
@@ -313,7 +298,7 @@ domains without causing confusion.
 Rose Config Files
 -----------------
 
-Use ``rose config-dump`` to load and re-save new Rose .conf files. This
+Use ``rose config-dump`` to load and re-save new ``rose.conf`` files. This
 puts the files in a standard format (ordering of lines etc.) to ensure that
 spurious changes aren't generated when you next use ``rose edit``.