Document how to write a new conformance test

and better explain the format of that file
and better how to invoke ./run-tests.sh

Author: mr-c

CONFORMANCE_TESTS.md

@@ -15,35 +15,153 @@ All tests passed
 
 ## Options
 
-RUNNER=other-cwl-runner
+`RUNNER=other-cwl-runner`
 
 The CWL implementation to be tested.
 
--nN
+### Test Selection
 
-Run a single specific test number N.
+`--tags`
 
-EXTRA=--parallel
+A comma separated list of [tags](#tags-for-conformance-tests); only tests with these tags will be tested.
+`--tags shell_command` will run all tests with `shell_command` in their `tags` list.
 
-Extra options to pass to the CWL runner
+`-n{test_range}`
 
-For example, to run conformance test 15 against the "cwltool"
-reference implementation with `--parallel`:
+Run only the specific test numbers. `{test_range}` is a comma separated list of
+single test numbers and/or numeric ranges.
+`-n5-7,15` == only runs the 5th, 6th, 7th, and 15th tests.
+
+`-N{test_range}`
+
+Like the lowercase `-n` option, except that the specified tests will not be run.
+Can be mixed with the other test selectors: `-n5-7,15 -N6` == only runs the 5th, 7th, and 15th tests, skipping the 6th test.
+
+`-s{test_names}`
+
+Run the specific tests according to their `label`s. `{test_names}` is a comma separated list of
+test labels. `-scl_optional_bindings_provided,stdout_redirect_docker,expression_any_null`
+achieves the same effect as `-n5-7,15 -N6` and it will still work if the tests are re-ordered.
+
+`-S{test_names}`
+
+Excludes specific tests according to their `label`s. `{test_names}` is a comma separated list of
+test labels. `--tags shell_command -Sstderr_redirect_shortcut` will run all test with `expression_tool`
+in their `tags` list except the test with the label `stderr_redirect_shortcut`.
+
+### Misc
+
+`EXTRA="--parallel --singularity"`
+
+Extra options to pass to the CWL runner (check your runner for exact options)
+
+`-j=4`
+
+The number of different tests to run at the same time
+
+`--junit-xml=FILENAME`
+
+Store results in JUnit XML format using the given FILENAME.
+
+`--classname=CLASSNAME`
+
+In the JUnit XML, tag the results with the given CLASSNAME.
+Can be useful when multiple test runs are combined to document the differences in `EXTRA=`.
+
+---
+
+For example, to run conformance test 15,16,17, and 20 against the "cwltool"
+reference implementation using the `podman` container engine
+and parallel execution of workflow steps
 
 ```bash
-$ ./run_test.sh RUNNER=cwltool -n15 EXTRA=--parallel
+$ ./run_test.sh RUNNER=cwltool -n15-17,20 EXTRA="--parallel --podman"
 Test [15/49]
+Test [16/49]
+Test [17/49]
+Test [20/49]
 All tests passed
 ```
 
-## Notes
+## OS X / macOS Notes
 
 _NOTE_: For running on OSX systems, you'll need to install coreutils via brew. This will add to your
 system some needed GNU-like tools like `greadlink`.
 
 1. If you haven't already, install [brew](http://brew.sh/) package manager in your mac
 2. Run `brew install coreutils`
 
+## Format of the conformance test file
+
+Conformance tests consist of input CWL descriptions plus an input CWL object
+and the expected outputs (or `should_fail: true` if the test is deliberately broken)
+
+They are stored in [`conformance_tests.yaml`](https://github.com/common-workflow-language/cwl-v1.2/blob/main/conformance_tests.yaml)
+(or a file `$import`ed into that one)
+
+You can examine [the formal schema of this file](https://github.com/common-workflow-language/cwltest/blob/main/cwltest/cwltest-schema.yml),
+or just continue reading here for an explanation.
+
+The conformance test file is a YAML document: a list of key-value pairs.
+
+We will use this single entry to explain the format
+``` yaml
+- doc: Test command line with optional input (missing)
+   label: cl_optional_inputs_missing
+   tool: tests/cat1-testcli.cwl  
+   job: tests/cat-job.json
+   output:
+     args: [cat, hello.txt]
+   tags: [ required, command_line_tool ]
+```
+- `doc`: A sentence that explain what is being tested. Will be printed at test execution time. Should be unique.
+- `label`: a short list of  underscore (`_`) separated words that succinctly identifies and explains the test.
+- `tool` the path to the CWL description to run
+- `job`: the CWL input object in YAML/JSON format. If there are no inputs then use `tests/empty.json`.
+- `output` [the CWL output object expected.](#output-matching)
+- `tags`: a yaml list of tag names, see [the list of canonical tags below](#tags-for-conformance-tests).
+
+Because this is a `schema-salad` processed document, `$import` can be used to organize the tests into separate files.
+
+Currently, the main file is too big (over 3400 lines); we are slowly re-organizing it.
+
+Eventually it would be good to organize the tests so that the test for each optional feature and other logical groups of tests are in their own separate file;
+with the supporting CWL documents and their inputs in separate sub-folders of `tests` as well.
+
+Example: [`- $import: tests/string-interpolation/test-index.yaml`](https://github.com/common-workflow-language/cwl-v1.2/blob/5f27e234b4ca88ed1280dedf9e3391a01de12912/conformance_tests.yaml#L3395)
+adds all the entries in [`tests/string-interpolation/test-index.yaml`](https://github.com/common-workflow-language/cwl-v1.2/blob/main/tests/string-interpolation/test-index.yaml)
+as entries in the main conformance test file.
+
+## Output matching
+
+In each test entry there is an `output` field that contains a mapping of the expected outputs names and their values.
+
+If a particular value could vary and it doesn't matter to the proper functioning of the test, then it can be represented by the special token `Any`.
+
+At any level, if there is an extra field, then that will be considered an error.
+An exception to this is `class: File` and `class: Directory` objects, the `cwl-runner` under test can add additional fields here.
+Likewise, if you don't want to test some aspect of a `class: File` or `class: Directory` object (like `nameext`) you can just omit it.
+
+[According to the CWL standards](https://www.commonwl.org/v1.2/CommandLineTool.html#File), the format of the `location` field in 
+`class: File` and `class: Directory` is implementation specific and we should not be testing them.
+Please remember to use `location: any` for them.
+
+Currently, we do [test the contents of the location field in some older tests, but we should stop](https://github.com/common-workflow-language/common-workflow-language/issues/930)
+If you are editing those old tests, you may be interested in some special processing for  `class: File` and `class: Directory` output objects:
+any `location` value specified will succeed if there is either an exact match to the real output, or it matches the end of the real output.
+Additionally, for `class: Directory` the location reported by the actual execution will have any trailing forward slash (`/`) trimmed off before comparison.
+
+## Writing a new conformance test
+
+To add a new conformance test:
+1. Ensure the CWL description you have tests the desired feature or aspect.
+2. Run your test using the CWL reference runner (`cwltool`) or another CWL runner
+     that shows the correct behavior to collect the output, or confirm that validation/execution fails as expected
+3. Add the CWL description and output object to the subdirectory `tests` in this repository.
+4. Fill out a new entry in [conformance_tests.yaml](conformance_tests.yaml) following the [format of the conformance test file](#format-of-the-conformance-test-file)
+5. Send a pull request to [current staging branch for the next revision of the CWL standards](https://github.com/common-workflow-language/cwl-v1.2/tree/1.2.1_proposed) 
+     with your changes
+
 ## Tags for conformance tests
 
 Each test in the [conformance_tests.yaml](conformance_tests.yaml) should be tagged with one or more tags.