Merge pull request #23 from UCL-ARC/1-unit-testing-feature-list

Add template testing feature matrix and update testing documentation

Author: connoraird

README.md

@@ -62,4 +62,7 @@ This will produce executables for the two src codes, `fortran-tooling-mesh-gener
 
 ## Running the tests
 
-To run the tests run `ctest` from within the `build` directory.
+To run the tests, from within the `build` directory, run the following.
+```sh
+$ ctest
+```

testing/README.md

@@ -11,10 +11,35 @@ There are several examples of good unit testing tools for other languages, such
 
 These will be used as the basis for what the recommended Fortran unit testing tool should look like. Therefore, key features from these tools shall be individually tested for each Fortran unit testing tool we select to test.
 
-## Aims
-- Test multiple fortran unit testing tools to determine which we would recommend. To do this we will
-    1. [ ] Create a list of features that should be present in a "good" unit testing tool.
-    2. [ ] Write tests to cover each of the key features determined in the previous step.
-        - [ ] [test-drive](./tests/test-drive)
-        - [ ] [pFUnit](./tests/pFUnit)
-- Build template implementations for the tools we recommend.
+## Running all of the tests
+
+All of the test can be ran from within the build directory with the command `ctest`. However, there is a known issue that the tests for test-drive itself will also be ran, as shown below.
+```sh
+$ ctest   
+Test project /Users/connoraird/work/fortran-tooling/build
+    Start 1: fortran-tooling-test-drive/mesh_generator
+1/4 Test #1: fortran-tooling-test-drive/mesh_generator ...   Passed    0.33 sec
+    Start 2: test-drive/all-tests
+2/4 Test #2: test-drive/all-tests ........................   Passed    0.33 sec
+    Start 3: test-drive/check
+3/4 Test #3: test-drive/check ............................   Passed    0.01 sec
+    Start 4: test-drive/select
+4/4 Test #4: test-drive/select ...........................   Passed    0.01 sec
+
+100% tests passed, 0 tests failed out of 4
+
+Total Test time (real) =   0.69 sec
+```
+
+## Template Features matrix
+
+Compilers tested: *A list of compilers we have tried with these tests*
+
+| Feature | Implemented natively | Implemented manually |
+|---------|----------------------|----------------------|
+| Can run individual tests | Yes or No (explanation) | Yes or No (explanation) |
+| Mocking | Yes or No (explanation) | Yes or No (explanation) |
+| Stubbing | Yes or No (explanation) | Yes or No (explanation) |
+| Data driven tests | Yes or No (explanation) | Yes or No (explanation) |
+| Coverage report | Yes or No (explanation) | Yes or No (explanation) |
+| Skip tests | Yes or No (explanation) | Yes or No (explanation) |

testing/test-drive/README.md

@@ -3,26 +3,14 @@ This project offers a lightweight, procedural unit testing framework based on no
 
 ## Running the tests
 
-The test-drive tests will run with the rest of the [tests](../README.md#running-the-tests) in the repo.
-
-There is a known issue that the tests for test-drive itself will also be ran by `ctest` as shown below
+All of the tests written with test-drive can be ran by following [#running all of the tests](../README.md#running-all-of-the-tests). However, we can also run a single testsuite or an individual test directly using the test executable. For example to run the `mesh_generator` tests we can run
 ```sh
-$ ctest   
-Test project /Users/connoraird/work/fortran-tooling/build
-    Start 1: fortran-tooling-test-drive/mesh_generator
-1/4 Test #1: fortran-tooling-test-drive/mesh_generator ...   Passed    0.33 sec
-    Start 2: test-drive/all-tests
-2/4 Test #2: test-drive/all-tests ........................   Passed    0.33 sec
-    Start 3: test-drive/check
-3/4 Test #3: test-drive/check ............................   Passed    0.01 sec
-    Start 4: test-drive/select
-4/4 Test #4: test-drive/select ...........................   Passed    0.01 sec
-
-100% tests passed, 0 tests failed out of 4
-
-Total Test time (real) =   0.69 sec
+$ <root of repo>/build/testing/test-drive/test_fortran-tooling-test-drive mesh_generator
+```
+and to run the single `mesh_generator` test `test_calculate_mesh_parameters_5_1`
+```sh
+$ <root of repo>/build/testing/test-drive/test_fortran-tooling-test-drive mesh_generator test_calculate_mesh_parameters_5_1
 ```
-
 
 ## Features matrix
 
@@ -33,9 +21,9 @@ Compilers tested: gfortran (homebrew)
 | Can run individual tests | No | Yes, see [main.f90](./main.f90). However, this requires running the test executable directly without ctest. |
 | Mocking | No | Not implemented |
 | Stubbing | No | Not implemented |
-| Data driven tests | No | Yes, see verify_calculate_mesh_parameters and verify_calculate_mesh in [test_mesh_generator.f90](./test_mesh_generator.f90)
+| Data driven tests | No | Yes, but this is very cumbersome. See `verify_calculate_mesh_parameters` and `verify_calculate_mesh` in [test_mesh_generator.f90](./test_mesh_generator.f90)
 | Coverage report | Yes, with fpm | N/A |
-| Skip tests | Yes, see test_skip_example in [test_calc_pi.f90](./test_mesh_generator.f90) | N/A |
+| Skip tests | Yes, see `test_skip_example` in [test_mesh_generator.f90](./test_mesh_generator.f90) | N/A |
 
 ## Pros
 - Lightweight, procedural unit testing framework based on nothing but standard Fortran.