Skip to content

Document logging of build steps in the outcome file #9934

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: development
Choose a base branch
from
Open

Document logging of build steps in the outcome file #9934

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c3d00b0
Outcome file: clarify documentation of some fields
gilles-peskine-arm Jan 25, 2025
f1133d6
Outcome file: document make and whole-component targets
gilles-peskine-arm Jan 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions docs/architecture/testing/test-framework.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,14 @@ The outcome file is in a CSV format using `;` (semicolon) as the delimiter and n

The outcome file has 6 fields:

* **Platform**: a description of the platform, e.g. `Linux-x86_64` or `Linux-x86_64-gcc7-msan`.
* **Configuration**: a unique description of the configuration (`mbedtls_config.h`).
* **Test suite**: `test_suite_xxx`, `ssl-opt` or `compat`.
* **Platform**: a description of the build platform, e.g. `Linux-x86_64`. This is normally `uname -sm` with punctuation tweaked, as set through the environment variable `MBEDTLS_TEST_PLATFORM` in `all.sh`.
* **Configuration**: a unique description of all build-time settings, including the library configuration (`mbedtls_config.h`, `crypto_config.h`) as well as compiler and linker flags. This defaults to the component name in `all.sh`, but should be made unique in components that build more than one configuration.
* **Test suite**: generally, this is the base name of the file containing the test case. This includes:
* `all.sh` components: `all.sh` (we do not keep track of which file a component is defined in). The “test case” column contains the component name.
* Build steps: `make` (a build target often involves multiple makefiles, so this is not traced to a specific makefile). The “test case” column contains the target.
* Unit test suites: `test_suite_xxx` (base name of the `.data` file, without the `.data` extension).
* TLS tests (`tests/ssl-opt.sh`): either `ssl-opt` or the name of a file in `tests/opt-testcases` without the `.sh` extension.
* TLS compatibility (`tests/compat.sh`): `compat`.
* **Test case**: the description of the test case.
* **Result**: one of `PASS`, `SKIP` or `FAIL`.
* **Cause**: more information explaining the result.