#3856: Use site-specific lists of recipes in the RTW

Co-authored-by: Emma Hogan <[email protected]>

Author: chrisbillowsMO

doc/sphinx/source/utils/RTW/add_a_recipe.rst

@@ -3,116 +3,141 @@ How to add a recipe to the |RTW|
 
 .. include:: common.txt
 
-.. note::
-   Before you follow these steps to add your recipe, you must be able to
-   successfully run the recipe with the latest version of ESMValTool on the
-   compute server you use at your site, as detailed by the ``platform`` option
-   in the ``[[COMPUTE]]`` section in the site-specific ``.cylc`` file in the
-   ``esmvaltool/utils/recipe_test_workflow/site/`` directory.
+Overview
+--------
 
-#. Open a `new ESMValTool issue`_ on GitHub, assign yourself to the issue, and
-   add the ``Recipe Test Workflow (RTW)`` label to the issue, see
-   `ESMValTool issue #3663`_ for an example.
+To add a recipe to the |RTW| you will:
 
-#. Create a branch.
+* Run the recipe at your site
+* Note the actual duration and memory usage
+* Edit your site's recipe file
+* Create the recipe's KGOs
+* Request a review
 
-#. Obtain the duration and memory usage of the recipe from the messages printed
-   to screen, or at the end of the ``run/main_log.txt`` file in the recipe
-   output directory after running your recipe on the compute cluster you use at
-   your site; these messages will look something like::
+The recipe will then run at your site whenever the |RTW| is run.
 
-     YYYY-MM-DD HH:MM:SS:sss UTC [12345] INFO    Time for running the recipe was: 0:02:13.334742
-     YYYY-MM-DD HH:MM:SS:sss UTC [12345] INFO    Maximum memory used (estimate): 2.4 GB
-     [...]
-     YYYY-MM-DD HH:MM:SS:sss UTC [12345] INFO    Run was successful
+Preparation
+-----------
 
-#. Add the recipe to the ``[task parameters]`` section in the
-   ``esmvaltool/utils/recipe_test_workflow/flow.cylc`` file.
+#. Open a `new ESMValTool issue`_ on GitHub. Assign yourself to the issue and
+   add the ``Recipe Test Workflow (RTW)`` label. `ESMValTool issue #3663`_
+   provides an example.
 
-   .. hint::
-      If the recipe takes less than 10 minutes to run then it should be added
-      to the ``fast`` option. Recipes that take longer than ten minutes should
-      be added to the ``medium`` option.
+#. Create a branch.
+
+#. Run the recipe:
+
+   * with the latest version of ESMValTool
+   * on the compute server you use at your site
 
    .. hint::
-      The line added should follow the format of ``recipe_new_recipe, \``,
-      unless the line is the last one in the list, in which case the line added
-      should follow the format of ``recipe_new_recipe``.
-
-#. If the duration of the recipe is larger than the value specified by the
-   ``execution time limit`` option in the ``[[COMPUTE]]`` section in the
-   aforementioned site-specific ``.cylc`` file, and / or the memory usage of
-   the recipe is larger than the value specified by the ``--mem`` option in the
-   ``[[[directives]]]`` section in the ``[[COMPUTE]]`` section, add a section
-   (in alphabetical order) to this file as shown below (round the duration to
-   the nearest second)::
-
-     [[process<fast=recipe_albedolandcover>]]
-     # Actual: 0m31s, 2.5 GB on 2024-04-08.
-     execution time limit = PT2M
-     [[[directives]]]
-         --mem = 3G
+      Your compute server is defined in the
+      ``esmvaltool/utils/recipe_test_workflow/site/<site>.cylc`` file as
+      follows::
+
+         [[COMPUTE]]
+            platform = <your compute server>
+
+#. Obtain the actual duration and memory usage of the recipe. This can be found
+   either in the message printed to screen, or at the end of the
+   ``run/main_log.txt`` file in the recipe output directory. The relevant lines will
+   look something like::
+
+      YYYY-MM-DD HH:MM:SS:sss UTC [12345] INFO    Time for running the recipe was: 0:02:13.334742
+      YYYY-MM-DD HH:MM:SS:sss UTC [12345] INFO    Maximum memory used (estimate): 2.4 GB
+      [...]
+      YYYY-MM-DD HH:MM:SS:sss UTC [12345] INFO    Run was successful
+
+Adding the recipe
+-----------------
+
+#. Add the recipe in alphabetical order to either ``FAST_RECIPES`` or
+   ``MEDIUM_RECIPES`` in the ``<site>-recipes.jinja`` file. It should look
+   something like::
+
+      {
+         'recipe_path': 'recipe_a_fast_recipe',
+         'actual': '2m13s, 2.4 GB on YYYY-MM-DD',
+         'max_time': 'PT3M',
+         'max_memory': '3G',
+      }
+
+   .. important::
+      Add the recipe to ``FAST_RECIPES`` if it takes *less* than 10 mins to
+      run at your site. Add the recipe to ``MEDIUM_RECIPES`` if it takes *more*
+      than 10 mins.
 
    .. hint::
-      The ``fast`` key in the example task definition above
-      (``[[process<fast=recipe_albedolandcover>]]``) should match name of the
-      option the recipe was added to in the ``[task parameters]`` section in
-      the ``esmvaltool/utils/recipe_test_workflow/flow.cylc`` file
+      The :ref:`site/site-recipes.jinja <site_recipes_file>`.
+      file provides more information.
 
    .. hint::
-      Set the ``execution time limit`` to 10-20% more than the actual duration.
-      For actual durations of up to ``1m45s``, set the ``execution time limit``
-      to ``PT2M`` (2 minutes).
+      Set the ``max_time`` to 10-20% more than the actual duration. For actual
+      durations of up to ``1m45s``, set ``max_time`` to ``PT2M`` (2 minutes).
 
    .. hint::
       Try not to regularly waste more than 500 MiB in memory usage. Typically,
       rounding the actual memory usage up to the nearest integer is acceptable.
 
-#. Stop any running ``recipe_test_workflow`` workflows::
-
-    cylc stop recipe_test_workflow/*
+Create the |KGOs|
+-----------------
 
 #. Run the |RTW|, as detailed in the :ref:`quick_start_guide`; it is expected
    that the ``compare`` task will fail.
 
-#. Update the Known Good Outputs (|KGOs|):
+   .. important::
+      The ``compare`` task fails because the |KGOs| for the recipe do not yet
+      exist. This run of the |RTW| will generate the outputs that will be
+      used as |KGOs|.
 
-   * Recursively copy the recipe output directory (i.e.
-     ``recipe_<recipe>_<date>_<time>/``) from the
-     ``${HOME}/cylc-run/recipe_test_workflow/runN/share/cycle/<cycle>/``
-     directory to your site-specific |KGO| directory, as detailed by the
-     ``KGO_ROOT_PATH`` option in the site-specific ``.conf`` file in the
-     ``esmvaltool/utils/recipe_test_workflow/opt/`` directory::
+#. Recursively copy the recipe output directory
+   ``recipe_<recipe>_<date>_<time>/`` to your site-specific |KGO| directory::
 
-       cp -r ${HOME}/cylc-run/recipe_test_workflow/runN/share/cycle/<cycle>/recipe_<recipe>_<date>_<time> <KGO_ROOT_PATH>
+      cp -r ${HOME}/cylc-run/recipe_test_workflow/runN/share/cycle/<cycle>/recipe_<recipe>_<date>_<time> <KGO_ROOT_PATH>
 
-     .. note::
-        Cylc is typically configured such that
-        ``${HOME}/cylc-run/recipe_test_workflow/runN/share`` is a symbolic link
-        to a share directory located on a scratch disk; the recipe output
-        directory is not stored in ``${HOME}``.
+   .. hint::
+      ``<cycle>`` will look something like: ``20250101T0900Z``. The recipe output
+      directory will look something like: ``recipe_python_20250101_090000``
+
+   .. hint::
+      Find your site specific ``KGO_ROOT_PATH`` in the ``rose-suite-<site>.conf`` file
+      in the ``esmvaltool/utils/recipe_test_workflow/opt/`` directory.
+
+   .. note::
+      Cylc is typically configured such that
+      ``${HOME}/cylc-run/recipe_test_workflow/runN/share`` is a symbolic link
+      to a share directory located on a scratch disk; the recipe output
+      directory is not stored in ``${HOME}``.
 
-   * Enable write permissions for all users on the recipe output directory in
-     your site-specific |KGO| directory::
+#. Enable write permissions for all users on the recipe output directory in
+   your site-specific |KGO| directory::
 
-       chmod -R a+w <KGO_ROOT_PATH>/recipe_<recipe>_<date>_<time>
+      chmod -R a+w <KGO_ROOT_PATH>/recipe_<recipe>_<date>_<time>
 
 #. Stop any running ``recipe_test_workflow`` workflows::
 
-     cylc stop recipe_test_workflow/*
+      cylc stop recipe_test_workflow/*
 
 #. Run the |RTW| again, as detailed in the :ref:`quick_start_guide`; the
    ``compare`` task should now succeed.
 
-#. Add the recipe to the documentation:
+   .. important::
+      Unless you want the |RTW| to continue running every night, repeat the
+      previous step to stop the workflow.
 
-   * Add a URL for the recipe in ``doc/sphinx/source/utils/RTW/common.txt``
-     under the ``.. Links`` section in alphabetical order (follow the format
-     ``.. _<name>: <URL>``).
+Request a review
+----------------
 
-   * Add the recipe to the list of :ref:`currently_tested_recipes` in
-     alphabetical order (follow the format ``* `<name>`_``).
+#. Commit and push your changes.
 
-#. Commit and push your changes, create a PR, assign yourself to the PR, and
-   add the ``Recipe Test Workflow (RTW)`` label to the PR, see
+#. Create a PR. Assign yourself to the PR, and add the
+   ``Recipe Test Workflow (RTW)`` label to the PR. Refer to
    `ESMValTool PR #3664`_ for an example.
+
+   .. note::
+      Reviewers will automatically be assigned to your PR.
+
+Congratulations!
+----------------
+
+The recipe will now run at your site whenever the RTW is run.

doc/sphinx/source/utils/RTW/common.txt

@@ -14,20 +14,16 @@
 
 .. Links
 
-.. _Autoassess landsurface soilmoisture: https://docs.esmvaltool.org/en/latest/recipes/recipe_autoassess_landsurface_soilmoisture.html#available-recipes-and-diagnostics
-.. _Consecutive dry days: https://docs.esmvaltool.org/en/latest/recipes/recipe_consecdrydays.html
 .. _CMIP Documentation: https://www.wcrp-climate.org/wgcm-cmip
 .. _Cylc Documentation: https://cylc.github.io/cylc-doc/stable/html/index.html
+.. _Cylc ISO8601 Durations: https://cylc.github.io/cylc-doc/stable/html/tutorial/scheduling/datetime-cycling.html#iso8601-durations
+.. _Cylc Jinja2: https://cylc.github.io/cylc-doc/stable/html/user-guide/writing-workflows/jinja2.html
 .. _Cylc Review: https://cylc.github.io/cylc-doc/7.9.3/html/tutorial.html#viewing-suite-logs-in-a-web-browser-cylc-review
-.. _Ensclus: https://docs.esmvaltool.org/en/latest/recipes/recipe_ensclus.html
 .. _ESMValTool Documentation: https://esmvaltool.org/
 .. _ESMValTool issue #3663: https://github.com/ESMValGroup/ESMValTool/issues/3663
 .. _ESMValTool PR #3664: https://github.com/ESMValGroup/ESMValTool/pull/3664
-.. _Examples python: https://docs.esmvaltool.org/en/latest/recipes/recipe_examples.html
-.. _Heatwaves coldwaves: https://docs.esmvaltool.org/en/latest/recipes/recipe_heatwaves_coldwaves.html
-.. _Landcover - Albedo: https://docs.esmvaltool.org/en/latest/recipes/recipe_albedolandcover.html#landcover-albedo
+.. _Jinja2: https://jinja.palletsprojects.com/en/stable/
 .. _new ESMValTool issue: https://github.com/ESMValGroup/ESMValTool/issues/new
-.. _Ocean Atlantic Meridional Overturning Circulation (AMOC): https://docs.esmvaltool.org/en/latest/recipes/recipe_oceans.html#recipe-ocean-amoc-yml
-.. _Ocean multimap: https://docs.esmvaltool.org/en/latest/recipes/recipe_oceans.html#recipe-ocean-multimap-yml
-.. _Radiation budget: https://docs.esmvaltool.org/en/latest/recipes/recipe_radiation_budget.html
 .. _Rose Documentation: https://metomi.github.io/rose/doc/html/index.html
+.. _metoffice-recipes.cylc: https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/utils/recipe_test_workflow/site/metoffice-recipes.cylc
+.. _Slurm sbatch --mem: https://slurm.schedmd.com/sbatch.html#OPT_mem

doc/sphinx/source/utils/RTW/tested_recipes.rst

@@ -6,14 +6,13 @@ Currently tested recipes
 
 .. include:: common.txt
 
-The following currently tested recipes are available in the |RTW|:
+The recipes tested with the |RTW| at each site are defined in the following
+files:
 
-* `Autoassess landsurface soilmoisture`_
-* `Consecutive dry days`_
-* `Ensclus`_
-* `Examples python`_
-* `Heatwaves coldwaves`_
-* `Landcover - Albedo`_
-* `Ocean Atlantic Meridional Overturning Circulation (AMOC)`_
-* `Ocean multimap`_
-* `Radiation budget`_
+* `metoffice-recipes.cylc`_
+* dkrz-recipes.cylc **(COMING SOON!)**
+* jasmin-recipes.cylc **(COMING SOON!)**
+
+.. hint::
+   The :ref:`site/site-recipes.jinja <site_recipes_file>`.
+   file provides more information.

doc/sphinx/source/utils/RTW/user_guide/workflow.rst

@@ -74,6 +74,17 @@ Portability
 The |RTW| is portable; site-specific information can be found in the ``site``
 and ``opt`` directories within the |RTW|. The files required are:
 
+.. _site_recipes_file:
+
+``site/<site>-recipes.jinja``
+   Contains all the recipes run at the ``SITE``
+
+.. hint::
+   * The file uses the `Jinja2`_ templating language, which has a similar syntax
+     to Python
+   * Jinja2 gives |Cylc| many powerful features. Refer to `Cylc Jinja2`_ for
+     more information
+
 ``site/<site>.cylc``
   Contains task definitions specific to the ``SITE``, for example, ``COMPUTE``
 

esmvaltool/utils/recipe_test_workflow/Jinja2Tests/file_exists.py

@@ -0,0 +1,37 @@
+"""
+Module for custom Jinja2 Tests.
+
+In Jinja2 "tests" are used to test a variable against an expression.
+Refer to Jinja2 tests:
+https://jinja.palletsprojects.com/en/stable/templates/#tests
+
+Jinja2 has a number of Builtin tests:
+https://jinja.palletsprojects.com/en/stable/templates/#list-of-builtin-tests
+
+Jinja2 also allows for writing custom tests in Python:
+https://jinja.palletsprojects.com/en/stable/api/#custom-tests
+
+Cylc supports custom filters, tests and globals as described here:
+https://cylc.github.io/cylc-doc/stable/html/user-guide/writing-workflows/jinja2.html#custom-jinja2-filters-tests-and-globals
+
+"""
+from pathlib import Path
+
+
+def file_exists(file_path):
+    """
+    Test if a file exists.
+
+    Parameters
+    ----------
+    file_path : str
+        A file path.
+
+    Returns
+    -------
+    bool
+        Returns True if the file exists.
+    """
+    run_directory = Path.cwd()
+    site_recipes_file_path = run_directory / file_path
+    return site_recipes_file_path.is_file()