From c3d00b03acc26cf5f4562d1d7742ce1d811e2286 Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Sat, 25 Jan 2025 18:13:37 +0100 Subject: [PATCH 1/2] Outcome file: clarify documentation of some fields Update some text from the early design that wasn't fully accurate. Signed-off-by: Gilles Peskine --- docs/architecture/testing/test-framework.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/architecture/testing/test-framework.md b/docs/architecture/testing/test-framework.md index a9e3dac47e1e..afcf856d753a 100644 --- a/docs/architecture/testing/test-framework.md +++ b/docs/architecture/testing/test-framework.md @@ -56,9 +56,12 @@ The outcome file is in a CSV format using `;` (semicolon) as the delimiter and n The outcome file has 6 fields: -* **Platform**: a description of the platform, e.g. `Linux-x86_64` or `Linux-x86_64-gcc7-msan`. -* **Configuration**: a unique description of the configuration (`mbedtls_config.h`). -* **Test suite**: `test_suite_xxx`, `ssl-opt` or `compat`. +* **Platform**: a description of the build platform, e.g. `Linux-x86_64`. This is normally `uname -sm` with punctuation tweaked, as set through the environment variable `MBEDTLS_TEST_PLATFORM` in `all.sh`. +* **Configuration**: a unique description of all build-time settings, including the library configuration (`mbedtls_config.h`, `crypto_config.h`) as well as compiler and linker flags. This defaults to the component name in `all.sh`, but should be made unique in components that build more than one configuration. +* **Test suite**: generally, this is the base name of the file containing the test case. This includes: + * Unit test suites: `test_suite_xxx` (base name of the `.data` file, without the `.data` extension). + * TLS tests (`tests/ssl-opt.sh`): either `ssl-opt` or the name of a file in `tests/opt-testcases` without the `.sh` extension. + * TLS compatibility (`tests/compat.sh`): `compat`. * **Test case**: the description of the test case. * **Result**: one of `PASS`, `SKIP` or `FAIL`. * **Cause**: more information explaining the result. From f1133d645cca0f080b8f59a866ab7efe3e881d48 Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Sat, 25 Jan 2025 18:26:12 +0100 Subject: [PATCH 2/2] Outcome file: document make and whole-component targets Signed-off-by: Gilles Peskine --- docs/architecture/testing/test-framework.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/architecture/testing/test-framework.md b/docs/architecture/testing/test-framework.md index afcf856d753a..7c20d309b2cb 100644 --- a/docs/architecture/testing/test-framework.md +++ b/docs/architecture/testing/test-framework.md @@ -59,6 +59,8 @@ The outcome file has 6 fields: * **Platform**: a description of the build platform, e.g. `Linux-x86_64`. This is normally `uname -sm` with punctuation tweaked, as set through the environment variable `MBEDTLS_TEST_PLATFORM` in `all.sh`. * **Configuration**: a unique description of all build-time settings, including the library configuration (`mbedtls_config.h`, `crypto_config.h`) as well as compiler and linker flags. This defaults to the component name in `all.sh`, but should be made unique in components that build more than one configuration. * **Test suite**: generally, this is the base name of the file containing the test case. This includes: + * `all.sh` components: `all.sh` (we do not keep track of which file a component is defined in). The “test case” column contains the component name. + * Build steps: `make` (a build target often involves multiple makefiles, so this is not traced to a specific makefile). The “test case” column contains the target. * Unit test suites: `test_suite_xxx` (base name of the `.data` file, without the `.data` extension). * TLS tests (`tests/ssl-opt.sh`): either `ssl-opt` or the name of a file in `tests/opt-testcases` without the `.sh` extension. * TLS compatibility (`tests/compat.sh`): `compat`.