From 2c09c9f62ab20880f01be8afec126fd6d76c1cb3 Mon Sep 17 00:00:00 2001 From: Lukasz Mrugala Date: Mon, 28 Oct 2024 09:29:05 +0000 Subject: [PATCH] doc: Add information on Twister Statuses Currently, Twister Statuses are only useful for the initiated, save for the very basics. This is in opposition to the fact that they are the main thing end users take into account when checking their Twister run reports. In order to make Statuses more useful for the end user, a new documentation page has been created, elucidating the full meaning of all Statuses. Signed-off-by: Lukasz Mrugala --- doc/develop/test/index.rst | 1 + doc/develop/test/twister.rst | 8 +- doc/develop/test/twister_statuses.rst | 215 ++++++++++++++++++++++++++ 3 files changed, 222 insertions(+), 2 deletions(-) create mode 100644 doc/develop/test/twister_statuses.rst diff --git a/doc/develop/test/index.rst b/doc/develop/test/index.rst index 9f43f86f8bc86..c984596e281df 100644 --- a/doc/develop/test/index.rst +++ b/doc/develop/test/index.rst @@ -8,6 +8,7 @@ Testing ztest twister + twister_statuses twister/twister_blackbox pytest coverage diff --git a/doc/develop/test/twister.rst b/doc/develop/test/twister.rst index f535648dd6e91..eb34435dded9f 100644 --- a/doc/develop/test/twister.rst +++ b/doc/develop/test/twister.rst @@ -19,8 +19,10 @@ complete code tree buildable. When using (at least) one ``-v`` option, twister's console output shows for every test application how the test is run (qemu, native_sim, etc.) or -whether the binary was just built. There are a few reasons why twister -only builds a test and doesn't run it: +whether the binary was just built. The resultant +:ref:`status ` +of a test is likewise reported in the ``twister.json`` and other report files. +There are a few reasons why twister only builds a test and doesn't run it: - The test is marked as ``build_only: true`` in its ``.yaml`` configuration file. @@ -214,6 +216,8 @@ env: used, for example, only if some required software or hardware is present, and to signal that presence to twister using these environment variables. +.. _twister_tests_long_version: + Tests ****** diff --git a/doc/develop/test/twister_statuses.rst b/doc/develop/test/twister_statuses.rst new file mode 100644 index 0000000000000..91e829564940b --- /dev/null +++ b/doc/develop/test/twister_statuses.rst @@ -0,0 +1,215 @@ +.. _twister_statuses: + +Twister Status +############## + +What is a Twister Status? +========================= + +Twister Status formulates the current state of + +- ``Harness`` +- ``TestCase`` +- ``TestSuite`` +- ``TestInstance`` + +in a comprehensive and easy-to understand way. +In practice, most users will be interested in Statuses +of Instances and Cases after the conclusion of their Twister runs. + +.. tip:: + + Nomenclature reminder: + + .. tabs:: + + .. tab:: ``Harness`` + + ``Harness`` is a Python class inside Twister that allows us + to capture and analyse output from a program external to Twister. + It has been excised from this page for clarity, + as it does not appear in final reports. + + .. tab:: ``TestCase`` + + ``TestCase``, also called Case, is a piece of code that aims to verify some assertion. + It is the smallest subdivision of testing in Zephyr. + + .. tab:: ``TestSuite`` + + ``TestSuite``, also called Suite, is a grouping of Cases. + One can modify Twister's behaviour on a per-Suite basis via ``testcase.yaml`` files. + Such grouped Cases should have enough in common + for it to make sense to treat them all the same by Twister. + + .. tab:: ``TestInstance`` + + ``TestInstance``, also called Instance, is a Suite on some platform. + Twister typically reports its results for Instances, + despite them being called "Suites" there. + If a status is marked as applicable for Suites, it is also applicable for Instances. + As the distinction between them is not useful in this section, + whenever you read "Suite", assume the same for Instances. + + More detailed explanation can be found :ref:`here `. + +Possible Twister Statuses +========================= + +.. list-table:: Twister Statuses + :widths: 10 10 66 7 7 + :header-rows: 1 + + * - In-code + - In-text + - Description + - Suite + - Case + * - FILTER + - filtered + - A static or runtime filter has eliminated the test from the list of tests to use. + - ✓ + - ✓ + * - NOTRUN + - not run + - The test wasn't run because it was not runnable in current configuration. + It was, however, built correctly. + - ✓ + - ✓ + * - BLOCK + - blocked + - The test was not run because of an error or crash in the test suite. + - ✕ + - ✓ + * - SKIP + - skipped + - Test was skipped by some other reason than previously delineated. + - ✓ + - ✓ + * - ERROR + - error + - The test produced an error in running the test itself. + - ✓ + - ✓ + * - FAIL + - failed + - The test produced results different than expected. + - ✓ + - ✓ + * - PASS + - passed + - The test produced results as expected. + - ✓ + - ✓ + +**In-code** and **In-text** are the naming contexts of a given status: +the former is rather internal for Twister and appears in logs, +whereas the latter is used in the JSON reports. + +.. note:: + + There are two more Statuses that are internal to Twister. + ``NONE`` (A starting status for Cases and Suites) and + ``STARTED`` (Indicating an in-progress Case). + Those should not appear in the final Twister report. + Appearance of those non-terminal Statuses in one's report files indicates a problem with Twister. + + +Case and Suite Status combinations +============================================ + +.. list-table:: Case and Suite Status combinations + :widths: 22 13 13 13 13 13 13 + :align: center + :header-rows: 1 + :stub-columns: 1 + + * - ↓ Case\\Suite → + - FILTER + - ERROR + - FAIL + - PASS + - NOTRUN + - SKIP + * - FILTER + - ✓ + - ✕ + - ✕ + - ✕ + - ✕ + - ✕ + * - ERROR + - ✕ + - ✓ + - ✕ + - ✕ + - ✕ + - ✕ + * - BLOCK + - ✕ + - ✓ + - ✓ + - ✕ + - ✕ + - ✕ + * - FAIL + - ✕ + - ✓ + - ✓ + - ✕ + - ✕ + - ✕ + * - PASS + - ✕ + - ✓ + - ✓ + - ✓ + - ✕ + - ✕ + * - NOTRUN + - ✕ + - ✕ + - ✕ + - ✕ + - ✓ + - ✕ + * - SKIP + - ✕ + - ✓ + - ✓ + - ✓ + - ✕ + - ✓ + +✕ indicates that such a combination should not happen in a proper Twister run. In other words, +no Suite of a status indicated by the table column should contain any Cases of a status indicated +by the table row. + +✓ indicates a proper combination. + +Detailed explanation, per Suite Status +------------------------------------------- + +``FILTER``: + This status indicates that the whole Suite has been statically filtered + out of a given Twister run. Thus, any Case within it should also have such a status. + +``ERROR``: + Suite encountered a problem when running the test. It requires at least one case with + ``ERROR`` or ``BLOCK`` status. As this takes precedence over all other Case statuses, all valid + terminal Case statuses can be within such a Suite. + +``FAIL``: + Suite has at least one Case that did not meet its assertions. This takes precedence over + all other Case statuses, given that the conditions for an ERROR status have not been met. + +``PASS``: + Suite has passed properly. It cannot contain any Cases with ``BLOCK``, ``ERROR``, or ``FAIL`` + statuses, as those indicate a problem when running the Suite. + +``NOTRUN``: + Whole suite was not run, but only built. It requires than all Cases within were not run. + As runnability is decided on a per-Suite basis, only ``NOTRUN`` is applicable for its Cases. + +``SKIP``: + Whole Suite has been skipped at runtime. All Cases need to have ``SKIP`` status as well.