twister: docs: Update twister docs with quarantine

Docs updated with changes proposed in #52179
Signed-off-by: Grzegorz Chwierut <[email protected]>

Author: gchwier

doc/develop/test/twister.rst

@@ -902,14 +902,15 @@ using an external J-Link probe.  The "probe_id" keyword overrides the
 Quarantine
 ++++++++++
 
-Twister allows using user-defined yaml files defining the list of tests to be put
-under quarantine. Such tests will be skipped and marked accordingly in the output
-reports. This feature is especially useful when running larger test suits, where
-a failure of one test can affect the execution of other tests (e.g. putting the
-physical board in a corrupted state).
+Twister allows user to provide onfiguration files defining a list of tests or
+platforms to be put under quarantine. Such tests will be skipped and marked
+accordingly in the output reports. This feature is especially useful when
+running larger test suits, where a failure of one test can affect the execution
+of other tests (e.g. putting the physical board in a corrupted state).
 
 To use the quarantine feature one has to add the argument
 ``--quarantine-list <PATH_TO_QUARANTINE_YAML>`` to a twister call.
+Multiple quarantine files can be used.
 The current status of tests on the quarantine list can also be verified by adding
 ``--quarantine-verify`` to the above argument. This will make twister skip all tests
 which are not on the given list.
@@ -920,22 +921,33 @@ to put under quarantine. In addition, an optional entry "comment" can be used, w
 some more details can be given (e.g. link to a reported issue). These comments will also
 be added to the output reports.
 
+When quarantining a class of tests or many scenarios in a single testsuite or
+when dealing with multiple issues within a subsystem, it is possible to use
+regular expressions, for example, **kernel.*** would quarantine
+all kernel tests.
+
 An example of entries in a quarantine yaml::
 
     - scenarios:
         - sample.basic.helloworld
-      platforms:
-        - all
       comment: "Link to the issue: https://github.com/zephyrproject-rtos/zephyr/pull/33287"
 
     - scenarios:
         - kernel.common
-        - kernel.common.misra
+        - kernel.common.(misra|tls)
         - kernel.common.nano64
       platforms:
-        - qemu_cortex_m3
+        - .*_cortex_.*
         - native_posix
 
+To exclude a platform, use the following syntax::
+
+    - platforms:
+      - qemu_x86
+      comment: "broken qemu"
+
+Additionally you can quarantine entire architectures or a specific simulator for executing tests.
+
 Running in Tests in Random Order
 ********************************
 Enable ZTEST framework's :kconfig:option:`CONFIG_ZTEST_SHUFFLE` config option to

scripts/pylib/twister/twisterlib/quarantine.py

@@ -40,7 +40,7 @@ class QuarantineElement:
     platforms: list[str] = field(default_factory=list)
     architectures: list[str] = field(default_factory=list)
     simulations: list[str] = field(default_factory=list)
-    comment: str = 'under quarantine'
+    comment: str = 'NA'
 
     def __post_init__(self):
         # If there is no entry in filters then take all possible values.

scripts/pylib/twister/twisterlib/testplan.py

@@ -753,7 +753,7 @@ def apply_filters(self, **kwargs):
                         instance.testsuite.id, plat.name, plat.arch, plat.simulation
                     )
                     if matched_quarantine and not self.options.quarantine_verify:
-                        instance.add_filter(matched_quarantine, Filters.QUARENTINE)
+                        instance.add_filter("Quarantine: " + matched_quarantine, Filters.QUARENTINE)
                     if not matched_quarantine and self.options.quarantine_verify:
                         instance.add_filter("Not under quarantine", Filters.QUARENTINE)
 

scripts/schemas/twister/quarantine-schema.yaml

@@ -25,19 +25,19 @@ sequence:
         type: seq
         sequence:
           - type: str
-          - unique: True
+          - unique: true
       "architectures":
         required: false
         type: seq
         sequence:
           - type: str
-          - unique: True
+          - unique: true
       "simulations":
         required: false
         type: seq
         sequence:
           - type: str
-          - unique: True
+          - unique: true
       "comment":
         type: str
         required: false

scripts/tests/twister/test_data/quarantines/platform.yaml

@@ -1,4 +1,3 @@
-
 - platforms:
   - demo_board_3
   comment: "all on board_3"

scripts/tests/twister/test_data/quarantines/with_regexp.yaml

@@ -1,4 +1,3 @@
-
 - scenarios:
   - test_(a|c).check_2
   architectures:

scripts/tests/twister/test_testplan_class.py

@@ -332,6 +332,6 @@ def test_quarantine(class_testplan, platforms_list, test_data,
         else:
             if testname in expected_val:
                 assert instance.status == 'filtered'
-                assert instance.reason == expected_val[testname]
+                assert instance.reason == "Quarantine: " + expected_val[testname]
             else:
                 assert not instance.status