Merge pull request #382 from oliver-sanders/migration-guide-2

728: migration guide

Author: oliver-sanders

src/7-to-8/cheat-sheet.rst

@@ -1,3 +1,5 @@
+.. _728.cheat_sheet:
+
 Cheat Sheet
 ===========
 

src/7-to-8/index.rst

@@ -14,9 +14,9 @@ Cylc 8 differs from Cylc 7 in many ways: architecture, scheduling
 algorithm, security, UIs, working practices, and more.
 
 .. toctree::
-   :maxdepth: 2
+   :maxdepth: 1
 
    summary
-   caveats
    cheat-sheet
    major-changes/index
+   caveats

src/7-to-8/major-changes/cli.rst

@@ -0,0 +1,160 @@
+Command Line Interface
+======================
+
+.. admonition:: Does This Change Affect Me?
+   :class: tip
+
+   This will affect you if you use the Cylc command line interface.
+
+
+Overview
+--------
+
+* Some commands have been renamed e.g. ``cylc run`` -> ``cylc play``.
+* Some tools have been added or removed.
+* The task ID format has changed.
+
+For a quick side by side comparison see the :ref:`728.cheat_sheet`.
+
+
+Full List Of Command Changes
+----------------------------
+
+The command line has been simplified from Cylc 7 with some commands being
+renamed or removed.
+
+.. _license: https://github.com/cylc/cylc-flow/blob/master/COPYING
+
+.. rubric:: Commands which have been removed or changed:
+
+``cylc checkpoint``
+  - Database checkpoints are no longer needed.
+  - All task state changes are written to the database when they occur.
+  - Remaining use cases can be handled by :term:`reflows <reflow>`
+    which allow a new execution of the graph to be started from an
+    arbitrary point in the graph.
+``cylc conditions``
+  - See the `license`_ file for conditions of usage
+  - The Cylc license remains unchanged from Cylc 7.
+``cylc documentation``
+  - We no longer include a command for locating this documentation.
+``cylc edit``
+  - Use a text editor to edit the workflow configuration file.
+``cylc insert``
+  - Task insertion is now automatic, use ``cylc trigger``.
+``cylc jobscript``
+  - It is no longer possible generate a jobscript from outside of a workflow.
+``cylc monitor``
+  - There is now a new more powerful terminal user interface (TUI).
+  - Try ``cylc tui``.
+``cylc nudge``
+  - No longer required.
+``cylc print``
+  - Equivalent to ``cylc scan --states=all``.
+``cylc register``
+  - Registration is no longer required, all workflows in the ``~/cylc-run``
+    directory are "registered" automatically.
+  - To install a workflow from a working copy use ``cylc install``.
+``cylc reset``
+  - It is no longer possible to manually change a task's state.
+  - You can, however, override the outputs the task generated which has a
+    similar effect with ``cylc set-outputs``.
+``cylc restart``
+  - ``cylc run``, ``cylc pause`` and ``cylc release`` have been combined into
+    ``cylc play``.
+``cylc review``
+  - The read-only ``cylc review`` web GUI has been removed.
+  - The latest Cylc 7 version of ``cylc review`` is Cylc 8 compatible
+    so can still be used to monitor both Cylc 7 and Cylc 8 workflows
+    side by side.
+``cylc run``
+  - ``cylc run``, ``cylc pause`` and ``cylc release`` have been combined into
+    ``cylc play``.
+``cylc search``
+  - Use ``grep`` or a text editor to search the workflow configuration or
+    source directory.
+``cylc spawn``
+  - Spawning is now performed automatically, use ``cylc trigger`` to run a task.
+``cylc submit``
+  - It is no longer possible to submit a job from outside of a workflow.
+``cylc warranty``
+  - The Cylc license remains unchanged from Cylc 7.
+
+.. rubric:: Graphical User Interfaces (GUIs):
+
+The GTK based GUI based GUIs have been removed, please use the new web based
+GUI. Consequently the following commands have also been removed:
+
+- ``cylc gpanel``
+- ``cylc gscan``
+- ``cylc gcylc``
+
+The ``cylc gui`` command remains, it launches a standalone version of the
+web GUI (providing the `Cylc UI Server`_ is installed).
+
+
+Cylc 8 Standardised IDs
+-----------------------
+
+In Cylc 7 there were two ways to specify a task:
+
+.. code-block:: none
+
+   task.cycle
+   cycle/task
+
+At Cylc 8 we have removed the former and extended the latter to provide a
+unique identifier for all workflows, cycles, tasks and jobs using a
+standardised format:
+
+.. code-block:: none
+
+   ~user/workflow//cycle/task/job
+
+Consequently task IDs have changed:
+
+.. code-block:: none
+
+   # old
+   cycle.task
+
+   # new
+   cycle/task
+
+An example using ``cylc trigger``:
+
+.. code-block:: bash
+
+   # old
+   cylc trigger workflow task.cycle
+
+   # new
+   cylc trigger workflow//cycle/task
+
+Cylc 8 still supports the old format, however, the new format unlocks extra
+functionality e.g:
+
+.. code-block:: bash
+
+   # stop all running workflows
+   cylc stop '*'
+   
+   # pause all running workflows
+   cylc pause '*'
+   
+   # (re-)trigger all failed tasks in all running workflows
+   cylc trigger '*//*:failed'
+   
+   # hold all tasks in the cycle "2000" in workflows with IDs
+    # beginning with "model"
+   cylc hold 'model*//2000'
+   
+   # delete the run directories for all workflows with IDs
+   # beginning with "model_a/"
+   cylc clean 'model_a/*'
+
+For more information run ``cylc help id``.
+
+.. _ID post on Discourse: https://cylc.discourse.group/t/cylc-8-id-changes/425
+
+For a quick overview of the motivation see the `ID post on Discourse`_.

src/7-to-8/major-changes/before-you-start.rst renamed to src/7-to-8/major-changes/compatibility-mode.rst

@@ -1,27 +1,56 @@
-.. _AutoConfigUpgrades:
+.. _cylc_7_compat_mode:
 
-Cylc Configuration Upgrader
-===========================
+Cylc 7 Compatibility Mode
+=========================
 
-.. admonition:: Does This Affect Me?
+.. admonition:: Does This Change Affect Me?
    :class: tip
 
-   This may affect you if you are working with workflows written before Cylc 8,
-   and you see warnings or an ``IllegalItemError`` when validating.
+   This will affect you if you want to run run Cylc 7 workflows (using the
+   ``suite.rc`` filename) using Cylc 8.0
 
 Overview
 --------
 
-Cylc has a built in configuration upgrader. **Cylc 8** can upgrade Cylc 7
-workflows on the fly if they use up-to-date Cylc 7 syntax, but *not
-if validation with Cylc 7 gives deprecation warnings*.
+The old ``suite.rc`` filename triggers a backward compatibility mode in which:
 
-Solution
---------
+- :term:`implicit tasks <implicit task>` are allowed by default
+
+  - (unless a ``rose-suite.conf`` file is found in the :term:`run directory`
+    for consistency with ``rose suite-run`` behaviour)
+  - (by default, Cylc 8 does not allow implicit tasks)
+
+- :term:`cycle point time zone` defaults to the local time zone
+
+  - (by default, Cylc 8 defaults to UTC)
+
+- waiting tasks are pre-spawned to mimic the Cylc 7 scheduling algorithm and
+  stall behaviour, and these require
+  :term:`suicide triggers <suicide trigger>`
+  for alternate :term:`graph branching`
+
+  - (Cylc 8 spawns tasks on demand and suicide triggers are not needed for
+    branching)
+
+- task ``succeeded`` outputs are *required* so the scheduler will retain
+  failed tasks as incomplete
+
+  - (in Cylc 8, all outputs are *required* unless marked as optional by new
+    ``?`` syntax)
+
+
+Required Changes
+----------------
+
+Providing your Cylc 7 workflow does not use syntax that was deprecated at Cylc 7
+you should be able to run it without any modifications.
+
+To see if your workflow uses deprecated syntax run ``cylc validate`` using
+Cylc 7.
+
+If tasks in your workflow call Cylc commands directly it might be necessary to
+modify them to be compatible with command line interface changes.
 
-To avoid problems with old settings you should validate your workflow using
-Cylc 7. Look for deprecation warnings and edit the workflow configuration to
-eliminate these warnings.
 
 Example
 -------
@@ -44,21 +73,21 @@ Running ``cylc validate`` on this configuration at **Cylc 7** we see that the
 workflow is valid, but we are warned that ``pre-command scripting``
 was replaced by ``pre-script`` at 6.4.0:
 
-.. code-block::
+.. code-block:: console
    :caption: Cylc 7 warning of a deprecated configuration
 
-   > cylc validate .
+   $ cylc validate .
    WARNING - deprecated items were automatically upgraded in 'suite definition':
    WARNING -  * (6.4.0) [runtime][task][pre-command scripting] -> [runtime][task][pre-script] - value unchanged
    Valid for cylc-7.8.7
 
 **Cylc 7** has upgraded this for us, but at **Cylc 8** this workflow will fail
 validation.
 
-.. code-block::
+.. code-block:: console
    :caption: Cylc 8 failing to validate an obsolete configuration
 
-   > cylc validate .
+   $ cylc validate .
    WARNING - deprecated graph items were automatically upgraded in "workflow definition":
     * (8.0.0) [scheduling][dependencies][X]graph -> [scheduling][graph]X - for X in:
           R1

src/7-to-8/major-changes/cylc-install.rst

@@ -44,17 +44,17 @@ Examples
 
 You can install from inside the development directory:
 
-.. code-block:: bash
+.. code-block:: console
 
-   > cd ~/cylc-src/my-workflow
-   > cylc install
+   $ cd ~/cylc-src/my-workflow
+   $ cylc install
    INSTALLED my-workflow/run1 from /home/me/cylc-src/my-workflow
 
 You can install by workflow name if the workflow is in ``~/cylc-src``.
 
-.. code-block:: bash
+.. code-block:: console
 
-   > cylc install my-workflow
+   $ cylc install my-workflow
    INSTALLED my-workflow/run2 from /home/me/cylc-src/my-workflow
 
 .. note::
@@ -64,6 +64,10 @@ You can install by workflow name if the workflow is in ``~/cylc-src``.
    ``cylc install`` also creates a symlink from the most recently installed run
    directory to ``~/cylc-run/<my_workflow>/runN``.
 
+.. tip::
+
+   Unwanted run directories can be removed with ``cylc clean``.
+
 You can also use ``-C`` (or ``--directory``) to set a source path:
 
 .. code-block:: bash
@@ -76,41 +80,54 @@ Once you have installed a workflow you can use ``cylc play`` to run it - see
 You can delete installed workflows using ``cylc clean`` - see
 :ref:`Removing-workflows`.
 
+A ``.cylcignore`` file can be used to control which files ``cylc install``
+transfers to the installed workflow, see :ref:`File Installation` for details.
+
+
+Remote Installation
+-------------------
+
+When the first task runs on a remote platform, Cylc will copy files from the
+:term:`run directory` to the remote platform.
 
-Cylc Rose
----------
+By default Cylc will transfer the ``app``, ``bin``, ``etc`` and ``lib``
+directories. This list is configurable see :ref:`RemoteInit` for more details.
+
+
+Rose Integration
+----------------
+
+The :ref:`Cylc Rose` plugin provides full support for
+:ref:`Rose suite configurations <Rose Suites>`.
 
 .. seealso::
 
-   :ref:`Cylc Rose` plugin documentation, which includes examples of how
-   to install a Cylc workflow with a Rose suite configuration with ``cylc install``.
+   * :ref:`Cylc Rose` plugin documentation.
+   * :ref:`installation` instructions.
 
-   .. TODO link to rose stem doc once it's done
 
-``cylc install`` is designed to allow the the use of configuration plugins.
-If you previously used ``rose suite-run`` you may wish to use the Cylc-Rose
-plugin.
+The Cylc Rose plugin runs automatically making ``cylc install`` a direct
+substitute for ``rose suite-run``.
 
 To find out if you have Cylc-Rose installed:
 
-.. code-block:: bash
+.. code-block:: console
 
-   > cylc version --long
+   $ cylc version --long
    8.0 (/path/to/cylc-8)
 
    Plugins:
        cylc-rose       0.1.1   /path/to/cylc-rose
 
-   ...
-
+Unlike ``rose suite-run``, the ``cylc install`` command remembers any options
+specified on the command line and preserves them for future re-installations.
 
+You may want to add ``~/roses`` to the list of
+:cylc:conf:`global.cylc[install]source dirs`.
 
-Rose Stem
----------
+Cylc Rose also provides the ``rose stem`` command which installs
+:ref:`rose-stem` suites. Once installed you can use ``cylc play`` to run them.
 
 .. seealso::
 
-   :ref:`rose-stem`
-
-The Cylc Rose plugin includes a ``rose stem`` command designed to allow the
-installation of Rose Stem suites. It is a small wrapper around ``cylc install``.
+   :ref:`Rose Stem` documentation.