Skip to content

Commit 7092335

Browse files
LukaszMrugalammahadevan108
LukaszMrugala
authored and
mmahadevan108
committed
doc: Add information on Twister Statuses
Currently, Twister Statuses are only useful for the initiated, save for the very basics. This is in opposition to the fact that they are the main thing end users take into account when checking their Twister run reports. In order to make Statuses more useful for the end user, a new documentation page has been created, elucidating the full meaning of all Statuses. Signed-off-by: Lukasz Mrugala <[email protected]>
1 parent 5f418f5 commit 7092335

File tree

3 files changed

+222
-2
lines changed

3 files changed

+222
-2
lines changed

‎doc/develop/test/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ Testing
88

99
ztest
1010
twister
11+
twister_statuses
1112
twister/twister_blackbox
1213
pytest
1314
coverage

‎doc/develop/test/twister.rst

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,10 @@ complete code tree buildable.
1919

2020
When using (at least) one ``-v`` option, twister's console output
2121
shows for every test application how the test is run (qemu, native_sim, etc.) or
22-
whether the binary was just built. There are a few reasons why twister
23-
only builds a test and doesn't run it:
22+
whether the binary was just built. The resultant
23+
:ref:`status <twister_statuses>`
24+
of a test is likewise reported in the ``twister.json`` and other report files.
25+
There are a few reasons why twister only builds a test and doesn't run it:
2426

2527
- The test is marked as ``build_only: true`` in its ``.yaml``
2628
configuration file.
@@ -214,6 +216,8 @@ env:
214216
used, for example, only if some required software or hardware is present, and to signal that
215217
presence to twister using these environment variables.
216218

219+
.. _twister_tests_long_version:
220+
217221
Tests
218222
******
219223

‎doc/develop/test/twister_statuses.rst

Lines changed: 215 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,215 @@
1+
.. _twister_statuses:
2+
3+
Twister Status
4+
##############
5+
6+
What is a Twister Status?
7+
=========================
8+
9+
Twister Status formulates the current state of
10+
11+
- ``Harness``
12+
- ``TestCase``
13+
- ``TestSuite``
14+
- ``TestInstance``
15+
16+
in a comprehensive and easy-to understand way.
17+
In practice, most users will be interested in Statuses
18+
of Instances and Cases after the conclusion of their Twister runs.
19+
20+
.. tip::
21+
22+
Nomenclature reminder:
23+
24+
.. tabs::
25+
26+
.. tab:: ``Harness``
27+
28+
``Harness`` is a Python class inside Twister that allows us
29+
to capture and analyse output from a program external to Twister.
30+
It has been excised from this page for clarity,
31+
as it does not appear in final reports.
32+
33+
.. tab:: ``TestCase``
34+
35+
``TestCase``, also called Case, is a piece of code that aims to verify some assertion.
36+
It is the smallest subdivision of testing in Zephyr.
37+
38+
.. tab:: ``TestSuite``
39+
40+
``TestSuite``, also called Suite, is a grouping of Cases.
41+
One can modify Twister's behaviour on a per-Suite basis via ``testcase.yaml`` files.
42+
Such grouped Cases should have enough in common
43+
for it to make sense to treat them all the same by Twister.
44+
45+
.. tab:: ``TestInstance``
46+
47+
``TestInstance``, also called Instance, is a Suite on some platform.
48+
Twister typically reports its results for Instances,
49+
despite them being called "Suites" there.
50+
If a status is marked as applicable for Suites, it is also applicable for Instances.
51+
As the distinction between them is not useful in this section,
52+
whenever you read "Suite", assume the same for Instances.
53+
54+
More detailed explanation can be found :ref:`here <twister_tests_long_version>`.
55+
56+
Possible Twister Statuses
57+
=========================
58+
59+
.. list-table:: Twister Statuses
60+
:widths: 10 10 66 7 7
61+
:header-rows: 1
62+
63+
* - In-code
64+
- In-text
65+
- Description
66+
- Suite
67+
- Case
68+
* - FILTER
69+
- filtered
70+
- A static or runtime filter has eliminated the test from the list of tests to use.
71+
- ✓
72+
- ✓
73+
* - NOTRUN
74+
- not run
75+
- The test wasn't run because it was not runnable in current configuration.
76+
It was, however, built correctly.
77+
- ✓
78+
- ✓
79+
* - BLOCK
80+
- blocked
81+
- The test was not run because of an error or crash in the test suite.
82+
- ✕
83+
- ✓
84+
* - SKIP
85+
- skipped
86+
- Test was skipped by some other reason than previously delineated.
87+
- ✓
88+
- ✓
89+
* - ERROR
90+
- error
91+
- The test produced an error in running the test itself.
92+
- ✓
93+
- ✓
94+
* - FAIL
95+
- failed
96+
- The test produced results different than expected.
97+
- ✓
98+
- ✓
99+
* - PASS
100+
- passed
101+
- The test produced results as expected.
102+
- ✓
103+
- ✓
104+
105+
**In-code** and **In-text** are the naming contexts of a given status:
106+
the former is rather internal for Twister and appears in logs,
107+
whereas the latter is used in the JSON reports.
108+
109+
.. note::
110+
111+
There are two more Statuses that are internal to Twister.
112+
``NONE`` (A starting status for Cases and Suites) and
113+
``STARTED`` (Indicating an in-progress Case).
114+
Those should not appear in the final Twister report.
115+
Appearance of those non-terminal Statuses in one's report files indicates a problem with Twister.
116+
117+
118+
Case and Suite Status combinations
119+
============================================
120+
121+
.. list-table:: Case and Suite Status combinations
122+
:widths: 22 13 13 13 13 13 13
123+
:align: center
124+
:header-rows: 1
125+
:stub-columns: 1
126+
127+
* - ↓ Case\\Suite →
128+
- FILTER
129+
- ERROR
130+
- FAIL
131+
- PASS
132+
- NOTRUN
133+
- SKIP
134+
* - FILTER
135+
- ✓
136+
- ✕
137+
- ✕
138+
- ✕
139+
- ✕
140+
- ✕
141+
* - ERROR
142+
- ✕
143+
- ✓
144+
- ✕
145+
- ✕
146+
- ✕
147+
- ✕
148+
* - BLOCK
149+
- ✕
150+
- ✓
151+
- ✓
152+
- ✕
153+
- ✕
154+
- ✕
155+
* - FAIL
156+
- ✕
157+
- ✓
158+
- ✓
159+
- ✕
160+
- ✕
161+
- ✕
162+
* - PASS
163+
- ✕
164+
- ✓
165+
- ✓
166+
- ✓
167+
- ✕
168+
- ✕
169+
* - NOTRUN
170+
- ✕
171+
- ✕
172+
- ✕
173+
- ✕
174+
- ✓
175+
- ✕
176+
* - SKIP
177+
- ✕
178+
- ✓
179+
- ✓
180+
- ✓
181+
- ✕
182+
- ✓
183+
184+
✕ indicates that such a combination should not happen in a proper Twister run. In other words,
185+
no Suite of a status indicated by the table column should contain any Cases of a status indicated
186+
by the table row.
187+
188+
✓ indicates a proper combination.
189+
190+
Detailed explanation, per Suite Status
191+
-------------------------------------------
192+
193+
``FILTER``:
194+
This status indicates that the whole Suite has been statically filtered
195+
out of a given Twister run. Thus, any Case within it should also have such a status.
196+
197+
``ERROR``:
198+
Suite encountered a problem when running the test. It requires at least one case with
199+
``ERROR`` or ``BLOCK`` status. As this takes precedence over all other Case statuses, all valid
200+
terminal Case statuses can be within such a Suite.
201+
202+
``FAIL``:
203+
Suite has at least one Case that did not meet its assertions. This takes precedence over
204+
all other Case statuses, given that the conditions for an ERROR status have not been met.
205+
206+
``PASS``:
207+
Suite has passed properly. It cannot contain any Cases with ``BLOCK``, ``ERROR``, or ``FAIL``
208+
statuses, as those indicate a problem when running the Suite.
209+
210+
``NOTRUN``:
211+
Whole suite was not run, but only built. It requires than all Cases within were not run.
212+
As runnability is decided on a per-Suite basis, only ``NOTRUN`` is applicable for its Cases.
213+
214+
``SKIP``:
215+
Whole Suite has been skipped at runtime. All Cases need to have ``SKIP`` status as well.

0 commit comments

Comments
 (0)