Merge pull request #37 from ricardogsilva/improve-docs

fleshing out standalone section of docs

Author: ricardogsilva

docs/development.md

@@ -27,7 +27,7 @@ into an instance of `models.TestSuiteResult`, its internal data structure.From
 there, it is able to serialize it into either JSON or markdown.
 
 
-### Setting up a develoment environment
+### Setting up a development environment
 
 In a brief nutshell:
 

docs/index.md

@@ -11,7 +11,7 @@ Key features:
 
 - **Runs as a github action**: Can be integrated into existing CI workflows
 - **Runs as a standalone CLI application**: Can be run as a standalone tool
-- **Multiple output formats**: Can output test suite results as markdown,
+- **Multiple output formats**: Can output test suite results as straight to the terminal in markdown,
   JSON or XML
 
 This project aims to simplify the running of OGC CITE tests suites so that

docs/running-as-standalone-cli.md

@@ -8,11 +8,18 @@ hide:
 
 ## Installation
 
-!!! note
+!!! info
 
     cite-runner will likely be available on the Python Package Index in the
     near future. That is not the case yet.
 
+    !!! warning
+
+        This also means that for the moment, cite-runner needs to be run through poetry, by means of using
+        `poetry run cite-runner`. As such, please remember to prefix all examples provided below with
+        `poetry run`.
+
+
 In order to use cite-runner locally you will need to have [git] and [poetry]
 installed. Once these are installed, cite-runner can be installed by cloning
 this repository and using poetry to install it:
@@ -29,49 +36,218 @@ You can verify the installation by running
 poetry run cite-runner --help
 ```
 
-##### Starting a local teamengine instance
 
-cite-runner is a [teamengine] runner. the OGC application used for running
-test suites. As such
+#### Starting a local teamengine instance
+
+cite-runner is a local runner for executing [OGC TEAM Engine] (aka teamengine). teamengine is the OGC application used
+for running test suites. As such, in order to use cite-runner, you also need to have an instance of
+teamengine at hand.
+
+One way of running teamengine is by pulling its [docker image] and
+running it locally. You can achieve this by running:
+
+```shell
+docker pull ogccite/teamengine-production:1.0-SNAPSHOT
+docker run \
+    --rm \
+    --name=teamengine \
+    --network=host \
+    ogccite/teamengine-production:1.0-SNAPSHOT
+```
+
+This will spawn a teamengine instance, which will be running locally on port `8080` - it will thus be accessible
+at:
+
+<http://localhost:8080/teamengine>
+
+!!! warning
+
+    cite-runner has been implemented to work with the teamengine version that is used in
+    the `ogccite/teamengine-production:1.0-SNAPSHOT` docker image as this is documented as
+    being the same version used in the OGC production system.
+
+    At the time of writing, this means **cite-runner is known to work with teamengine version
+    5.6.1**.
+
+    cite-runner has not been tested with other versions of teamengine.
 
 
-### Usage
+## Usage
 
-cite-runner is executed by calling the `cite-runner` command with a sub-command
-and suitable arguments. For example:
+Once installed, cite-runner can be executed by calling the `cite-runner` application with a command
+and suitable arguments.
 
 ```shell
-poetry run cite-runner execute-test-suite \
-    http://localhost:8080/teamengine \
-    ogcapi-features-1.0 \
-    --test-suite-input iut http://localhost:5000 \
-    --test-suite-input noofcollections -1
+cite-runner [OPTIONS] COMMAND [ARGS] ...
 ```
 
+!!! tip
 
-##### Sub-commands
+    The `--help` option can be used to discover how to use cite-runner interactively, like this:
 
-3. Start your service to be tested. Let's assume it is already running on `http://localhost:5000`
+    ```shell
+    cite-runner --help
+    ```
 
-4. Run the teamengine docker image locally:
+## Commands
 
-  ```shell
-  docker run \
-    --rm \
-    --name teamengine \
-    --network=host \
-    ogccite/teamengine-production:1.0-SNAPSHOT
-  ```
+### execute-test-suite
+
+Execute a teamengine CITE test suite and get its result.
+
+Results are serialized with the requested output format and are printed to the application's standard output stream,
+which is usually the terminal. If needed, you can redirect the output to a file.
+
+```shell
+cite runner execute-test-suite [OPTIONS] TEAMENGINE_BASE_URL TEST_SUITE_IDENTIFIER
+```
+
+
+##### Arguments
+
+- `TEAMENGINE_BASE_URL` - Base URL of the teamengine service. Example: `http://localhost:8080/teamengine`
+- `TEST_SUITE_IDENTIFIER` - Identifier of the test suite as known to teamengine. Example: `ogcapi-features-1.0`
+
+
+##### Options
+
+| name | description |
+| ---- | ----------- |
+| `--help` | Show information on how to run the command, including a description of arguments and options |
+| `--teamengine-username` | Username for authenticating with teamengine |
+| `--teamengine-password` | Password for authenticating with teamengine |
+| `--test-suite-input` | Inputs expected by teamengine for running the test suite specified with TEST_SUITE_IDENTIFIER. These vary depending on the test suite.<br><br>This parameter can be specified multiple times.<br><br>Each parameter must be specified as a name and a value, separated by the space character (_i.e._ `--test-suite-input {name} {value}`).<br><br>Example: `--test-suite-input iut http://localhost:5000 --test-suite-input noofcollections -1`|
+| `--output-format` | Format for the cite-runner result. Available options are:<br><ul><li><code>markdown</code> - Return results as a markdown document - This is the default</li><li><code>json</code> - Return results as JSON</li><li><code>console</code> - Return results in a format suitable for reading in the terminal</li><li><code>raw</code> - Return the raw results as provided by teamengine. This is an XML document</li></ul>
+| `--include-summary`/`--no-include-summary` | Whether the output should include a summary. This is enabled by default. Disable it by providing `--no-include-summary` |
+| `--include-failed-detail`/`--no-include-failed-detail` | Whether the output should include a section with details about failed tests. This is enabled by default, disable it by providing `--no-include-summary`.|
+| `--include-skipped-detail`/`--no-include-skipped-detail` | Whether the output should include a section with details about skipped tests. This is enabled by default, disable it by providing `--no-include-summary`.|
+| `--include-passed-detail`/`--no-include-passed-detail` | Whether the output should include a section with details about passed tests. This is disabled by default, enable it by providing `--include-summary`.|
+| `--exit-with-error-on-suite-failed-result`/`--no-exit-with-error-on-suite-failed-result` | Whether the application should exit with an error code when a suite is declared as not failed. This is disabled by default, enable it by providing `--exit-with-error-on-suite-failed-result`|
+
+
+##### Examples
+
+1. Run the test suite for OGC API Features, using a service that is running locally on port 5000 and then output just the
+   result summary to the terminal:
+
+    ```shell
+    cite-runner execute-test-suite \
+        http://localhost:8080/teamengine \
+        ogcapi-features-1.0 \
+        --test-suite-input iut http://localhost:5000 \
+        --test-suite-input noofcollections -1 \
+        --output-format console \
+        --no-include-skipped-detail \
+        --no-include-failed-detail
+    ```
+
+2. Run the test suite for OGC API Features using the [pygeoapi demo service] and then output the
+   full report in markdown format, redirecting the output to the `result.md` file:
+
+    ```shell
+    cite-runner execute-test-suite \
+        http://localhost:8080/teamengine \
+        ogcapi-processes-1.0 \
+        --test-suite-input iut https://demo.pygeoapi.io/stable \
+        --test-suite-input noofcollections -1 \
+        --include-passed-detail \
+    > result.md
+    ```
+
+3. Run the test suite for OGC API Processes using a service that is running locally on port 5000 and then output the
+   full report in JSON format, piping it to `jq` for further filtering:
+
+    ```shell
+    cite-runner execute-test-suite \
+        http://localhost:8080/teamengine \
+        ogcapi-processes-1.0 \
+        --test-suite-input iut http://localhost:5000 \
+        --test-suite-input noofcollections -1 \
+        --include-passed-detail \
+    | jq '.passed'
+    ```
+
+
+### parse-result
+
+Parse previously gotten results from an earlier cite-runner run that used `raw` as its output format.
+
+This command is most useful when you want to produce multiple reports in different output formats or with different
+details from the same test run.
+
+##### Arguments
+
+- `TEST_SUITE_RESULT` - Path to an XML file containing the raw execution results of a previous cite-runner run. You
+  can also use a raw result file generated by teamengine, as long as it is has been generated with
+  [teamengine EARL output format]
+
+
+##### Options
+
+Accepts a subset of similar [options as the execute-test-suite-command](#options)  namely:
+
+- `--output-format`
+- `--include-summary`/`--no-include-summary`
+- `--include-failed-detail`/`--no-include-failed-detail`
+- `--include-skipped-detail`/`--no-include-skipped-detail`
+- `--include-passed-detail`/`--no-include-passed-detail`
+- `--exit-with-error-on-suite-failed-result`/`--no-exit-with-error-on-suite-failed-result`
+
+
+##### Examples
+
+1. Parse a previously generated `raw-results.xml` and output results for consumption in the terminal:
+
+    ```shell
+    cite-runner parse-result raw-results.xml --output-format console
+    ```
+
+2. Run the OGC API Features test suite, then save the raw results in the `raw-results.xml` file and then parse
+   them into a markdown report
+
+    ```shell
+    cite-runner execute-test-suite \
+        http://localhost:8080/teamengine \
+        ogcapi-processes-1.0 \
+        --test-suite-input iut http://localhost:5000 \
+        --test-suite-input noofcollections -1 \
+        --output-format raw
+    > raw-results.xml
+
+    cite-runner parse-result raw-results.xml --include-passed-detail > parsed-results.md
+
+    ```
+
+
+### execute-test-suite-from-github-actions
+
+This command is merely a convenience for when executing cite-runner via github actions. When running cite-runner
+as a standalone tool you should prefer to use the [execute-test-suite command](#execute-test-suite) instead
+
+## Global options
+
+cite-runner includes a couple of global options. These are mainly useful for debugging. They must to be provided
+before the command.
+
+
+| name | description |
+| ---- | ----------- |
+| `--debug`/`--no-debug` | Whether to run cite-runner in debug mode or not. Debug mode provides additional runtime information, which can be used during development. This is disabled by default, enable it by providing `--debug`|
+| `--network-timeout` | How many seconds to use as the timeout parameter when contacting the teamengine service. The default value is `120` |
+
+##### Examples
+
+1. Run cite-runner in debug mode:
 
-- Run the action code with
+    ```shell
+    cite-runner --debug parse-result raw-results.xml --output-format console
+    ```
 
-  ```shell
-  poetry run cite-runner --help
-  ```
 
-There are additional commands and options which can be used when running locally, which allow controlling the output
-format and how the inputs are supplied. Read the online
 
+[docker image]: https://hub.docker.com/r/ogccite/teamengine-production
 [git]: https://git-scm.com/
 [poetry]: https://python-poetry.org/
-[teamengine]:
+[pygeoapi demo service]: https://demo.pygeoapi.io/stable
+[OGC TEAM Engine]: https://opengeospatial.github.io/teamengine/
+[teamengine EARL output format]: https://opengeospatial.github.io/teamengine/users.html#EARL_.28RDF.2FXML.29

src/cite_runner/config.py

@@ -73,9 +73,7 @@ def configure_logging(rich_console: Console, debug: bool) -> None:
 
 
 def get_console() -> Console:
-    return Console(
-        stderr=True,
-    )
+    return Console()
 
 
 def get_context(