Merge pull request #750 from oliver-sanders/document-workflow-log-files

document workflow log files

Author: oliver-sanders

src/7-to-8/summary.rst

@@ -236,6 +236,36 @@ Scheduling Algorithm
 The scheduling algorithm has been changed, more information is available:
 :ref:`728.scheduling_algorithm`.
 
+Log Files
+---------
+
+The workflow log files have moved to new locations and some new files have been
+added. For information on the Cylc 8 log files, see
+:ref:`user-guide.log_files`.
+
+.. list-table::
+
+   * - **Cylc 7** (and Rose 2019)
+     - **Cylc 8**
+   * - ``log/suite/log``
+     - ``log/scheduler/log``
+   * - ``log/suite/log.<time>``
+     - ``log/scheduler/<start_number>-<type>-<file_number>.log``
+   * - ``suite.rc.processed``
+     - ``log/config/flow-processed.cylc``
+   * - ``log/rose-suite-run.log``
+     - ``log/install/<start_number>-install.log``
+
+       ``log/remote-install/<start_number>-<type>-<platform>.log``
+   * - ``log/rose-conf/<time>-run.conf``
+     - ``log/config/<time>-rose-suite.conf``
+   * - ``log/<time>-run.version``
+     - ``log/version/uncommitted.diff``
+
+       ``log/version/vcs.json``
+   * - ``log/suiterc/<time>-run.rc``
+     - ``log/config/<start_number>-<type>-<file_number>.cylc``
+
 Other Changes
 -------------
 

src/dictionaries/words

@@ -198,11 +198,13 @@ templating
 th
 timeout
 timeouts
+timestamp
 timestep
 timezone
 timezones
 tuple
 uiserver
+unparsed
 unpaused
 unpausing
 untracked

src/user-guide/running-workflows/scheduler-log-files.rst

@@ -1,7 +1,113 @@
 .. _Scheduler Logs:
+.. _user-guide.log_files:
 
-Scheduler Logs
---------------
+Workflow Logs
+=============
+
+Cylc produces log files which record information about how the workflow was
+installed, configured when started or changed, and about tasks that have run
+and their outcomes.
+
+
+Log Files
+---------
+
+Cylc log files are located in the workflow :term:`run directory`.
+
+.. describe:: log/scheduler/
+
+   Contains the scheduler log files, these detail everything that a workflow
+   did whilst it ran and are key to debugging issues.
+
+   Every time you start or restart a workflow a new log file is created in this
+   location:
+
+   .. describe:: log/scheduler/<start_number>-<type>-<file_number>.log
+
+      For more information on this file, see :ref:`log_files.scheduler_log`.
+
+.. describe:: log/config/
+
+   Contains a record of the workflow configuration files:
+
+   .. describe:: log/config/<start_number>-<type>-<file_number>.cylc
+
+      The parsed contents of the ``flow.cylc`` (or ``suite.rc``) file from
+      each start, restart or reload of the workflow.
+
+   .. describe:: log/config/flow-processed.cylc
+
+      The unparsed contents of the ``flow.cylc`` (or ``suite.rc``) file that
+      the workflow was most recently started or restarted from. This file has
+      not been parsed by Cylc so is still in the definition order, however,
+      Jinja2 process has been performed making it useful for debugging Jinja2
+      issues.
+
+   .. describe:: log/config/<time>-rose-suite.conf
+
+      The parsed contents of the ``rose-suite.conf`` file if present in the workflow.
+
+.. describe:: log/install/
+
+   Contains the workflow installation log (i.e. the output of the
+   ``cylc install`` command.
+
+   .. describe:: log/install/<start_number>-install.log
+
+.. describe:: log/remote-install/
+
+   Contains a record of any remote platforms that the workflow has been
+   installed onto.
+
+   .. describe:: log/remote-install/<start_number>-<type>-<platform>.log
+
+.. describe:: log/version/
+
+   If the workflow :term:`source directory` is version controlled with Git or
+   SVN, this directory will contain information extracted from the version
+   control system at the time the workflow was installed:P
+
+   .. describe:: log/version/uncommitted.diff
+
+      Any uncommitted diff in unified diff format.
+
+   .. describe:: log/version/vcs.json
+
+      Information including the commit hash / revision number.
+
+.. describe:: opt/rose-suite-cylc-install.conf
+
+   If Rose is used, this file will contain any Rose options that were used
+   with the ``cylc install`` command (i.e. any overrides which were applied
+   when the workflow was installed).
+
+.. note::
+
+   Where the ``<variables>`` in the above paths are:
+
+   .. cylc-scope:: global.cylc[scheduler][logging]
+
+   ``start_number``
+      Is ``1`` when you first start the workflow. This number increments with
+      each restart.
+   ``type``
+      Is either ``start``, ``restart`` or ``reload``.
+   ``file_number``
+      Is initially ``1``, if the log file exceeds the `maximum size in bytes`,
+      it will "roll over" into a new log file and this number will increment.
+      If the number of "roll over" log files exceeds the
+      `rolling archive length`, then Cylc will remove an old log file before
+      creating a new one.
+   ``time``
+      Is a timestamp in ISO8601 format.
+
+.. cylc-scope::
+
+
+.. _log_files.scheduler_log:
+
+The Scheduler Log File
+----------------------
 
 Each workflow maintains its own log of time-stamped events in the
 :term:`workflow log directory` (``$HOME/cylc-run/<workflow-id>/log/scheduler/``).

src/user-guide/troubleshooting.rst

@@ -61,22 +61,7 @@ for debugging some specific issues:
    These files can allow you to see how the configuration was changed between
    restarts and reloads.
 
-   Each time a workflow is started, restarted or reloaded,
-   the configuration used is recorded in a file in ``log/config``.
-   This provides a series of snapshots of the configuration.
-   These files are named:
-
-   ``<LOG FILE NUMBER>-<EVENT>-<(RE)START NUMBER>``.
-
-   If for example we did:
-
-   .. code-block:: console
-
-      # Command            # File created
-      cylc play workflow   # 01-start-01.cylc
-      cylc vr workflow     # 02-reload-01.cylc
-      cylc stop workflow
-      cylc play workflow   # 03-restart-02.cylc
+For more information on workflow log files, see :ref:`user-guide.log_files`.
 
 
 Shell Login/Profile Files