Merge pull request #6900 from oliver-sanders/dev-docs++

improve dev docs

Author: oliver-sanders

tests/flakyfunctional/README.md

@@ -1,15 +1,15 @@
 # Flaky Functional Tests
 
-This directory contains tests that are sensitive to timing, server load, etc.
+This directory contains functional tests that are sensitive to timing, server
+load, etc.
+
+For more information on functional tests, see ../functional/README.md.
 
 ## How To Run These Tests
 
 ```console
 $ etc/bin/run-functional-tests tests/k
 
-# 4 tests in parallel
-$ etc/bin/run-functional-tests tests/k
-
 # split the tests into 4 "chunks" and run the first chunk
 $ CHUNK='1/4' etc/bin/run-functional-tests tests/k
 ```

tests/functional/README.md

@@ -2,13 +2,21 @@
 
 This directory contains Cylc functional tests.
 
+
 ## How To Run These Tests
 
 ```console
+# run all tests in this directory
 $ etc/bin/run-functional-tests tests/f
 
-# 4 tests in parallel
-$ etc/bin/run-functional-tests tests/f
+# run a specified test
+$ etc/bin/run-functional-tests tests/f/cylc-cat-log/00-local.t
+
+# run a specified test in debug mode
+$ etc/bin/run-functional-tests -v tests/f/cylc-cat-log/00-local.t
+
+# run tests with 5x parallelism
+$ etc/bin/run-functional-tests -j 5 tests/f
 
 # split the tests into 4 "chunks" and run the first chunk
 $ CHUNK='1/4' etc/bin/run-functional-tests tests/f
@@ -20,6 +28,7 @@ $ etc/bin/run-functional-tests tests/f
 $ coverage combine
 ```
 
+
 ## What Are Functional Tests?
 
 These tests ensure end-to-end functionality is as expected.
@@ -30,6 +39,48 @@ cause many parts of the system (and other systems) to be activated.
 This includes interaction with other systems (e.g. batch schedulers/job runners),
 command line interfaces / outputs, etc.
 
+
+## How Are Tests Implemented
+
+Tests are written in files with the `.t` extension. These are Bash scripts
+(despite the file extension). The tests are run with a tool called `prove`
+which is invoked via the `etc/bin/run-functional-tests` command.
+
+Each test file starts by sourcing the file `lib/bash/test_header`, this
+contains various Bash functions which provide assertion functions, e.g.
+`run_ok` (which asserts that the provided command succeeds).
+
+Assertions run a command, then write either `ok` or `not ok` to stdout, prove
+then scrapes this output to determine test outcome.
+
+Each assertion (or subtest) needs a name, this is usually the first argument
+to the function. You should use the prefix `$TEST_NAME_BASE`, e.g, this
+assertion tests that the command `sleep 10` succeeds
+`run_ok "${TEST_NAME_BASE}-sleep" sleep 10`. Note, some assertions infer the
+test name for you from the arguments.
+
+For the list of available assertions, and usage info, see the comments at the
+top of `lib/bash/test_header`.
+
+Each test file needs to declare the number of subtests within it, this is
+done with the function `set_test_number` which must be run at the start of the
+test.
+
+Many tests install workflows for testing. These are installed into cylc-run as
+usual under the prefix `cylctb`.
+
+
+## How To Debug Failing Tests
+
+* Run the test using the `-v` argument, this will reveal which subtest(s)
+  failed.
+* Some tests reveal extra debug info when you set `CYLC_TEST_DEBUG=true`.
+* If the test fails, any workflows installed into the `~/cylc-run/cylctb???`
+  directory will not be deleted, you can inspect the workflow logs there.
+* In GitHub actions, the `~/cylc-run` directory is uploaded as an "artifact"
+  where it can be downloaded for debugging.
+
+
 ## How To Run "Non-Generic" Tests?
 
 Some tests require job runners (e.g. at, slurm, pbs), remote platforms or
@@ -62,6 +113,7 @@ $ etc/bin/run-functional-tests -p '_*at*' tests/f
 $ etc/bin/run-functional-tests -p '*' tests/f
 ```
 
+
 ## Test Platform Names
 
 Each platform is named using this convention:
@@ -100,6 +152,7 @@ Define any test platforms in your global config e.g:
         hosts = my_remote_host
 ```
 
+
 ## Test Global Config
 
 Cylc supports a `global-tests.cylc` file which can be used to define some
@@ -108,6 +161,7 @@ top-level configurations to run tests with.
 Do not use this file to define test platforms, put them in your regular global
 config where they can also be used for interactive work.
 
+
 ## How To Configure "Non-Generic" Tests?
 
 By default tests require the platform `_local_background_indep_tcp`.
@@ -143,6 +197,7 @@ export REQUIRE_PLATFORM='runner:*'
 export REQUIRE_PLATFORM='loc:remote runner:*'
 ```
 
+
 ## Guidelines
 
 Don't write functional tests when you can write integration tests: