twister: ztest: short test case names on --no-detailed-test-id

doc/develop/test/twister.rst

@@ -261,8 +261,7 @@ A Test Suite is a collection of Test Cases which are intended to be used to test
 a software program to ensure it meets certain requirements. The Test Cases in a
 Test Suite are either related or meant to be executed together.
 
-The name of each Test Scenario needs to be unique in the context of the overall
-test application and has to follow basic rules:
+Test Scenario, Test Suite, and Test Case names must follow to these basic rules:
 
 #. The format of the Test Scenario identifier shall be a string without any spaces or
    special characters (allowed characters: alphanumeric and [\_=]) consisting
@@ -272,7 +271,8 @@ test application and has to follow basic rules:
    subsection names delimited with a dot (``.``). For example, a test scenario
    that covers semaphores in the kernel shall start with ``kernel.semaphore``.
 
-#. All Test Scenario identifiers within a ``testcase.yaml`` file need to be unique.
+#. All Test Scenario identifiers within a Test Configuration (``testcase.yaml`` file)
+   need to be unique.
    For example a ``testcase.yaml`` file covering semaphores in the kernel can have:
 
    * ``kernel.semaphore``: For general semaphore tests
@@ -295,6 +295,18 @@ test application and has to follow basic rules:
      Test Case name, for example: ``debug.coredump.logging_backend``.
 
 
+The ``--no-detailed-test-id`` command line option modifies the above rules in this way:
+
+#. A Test Suite name has only ``<Test Scenario identifier>`` component.
+   Its Application Project path can be found in ``twister.json`` report as ``path:`` property.
+
+#. With short Test Suite names in this mode, all corresponding Test Scenario names
+   must be unique for the Twister execution scope.
+
+#. **Ztest** Test Case names have only Ztest components ``<Ztest suite name>.<Ztest test name>``.
+   Its parent Test Suite name equals to the corresponding Test Scenario identifier.
+
+
 The following is an example test configuration with a few options that are
 explained in this document.
 

doc/releases/release-notes-4.1.rst

@@ -113,6 +113,17 @@ Build system and Infrastructure
   them can use the :zephyr_file:`scripts/utils/twister_to_list.py` script to
   automatically migrate Twister configuration files.
 
+* Twister
+
+  * Test Case names for Ztest now include Ztest suite name, so the resulting identifier has
+    three sections and looks like: ``<test_scenario_name>.<ztest_suite_name>.<ztest_name>``.
+    These extended identifiers are used in log output, twister.json and testplan.json,
+    as well as for ``--sub-test`` command line parameters (:github:`80088`).
+  * The ``--no-detailed-test-id`` command line option also shortens Ztest Test Case names excluding
+    its Test Scenario name prefix which is the same as the parent Test Suite id (:github:`82302`).
+    Twister XML reports have full testsuite name as ``testcase.classname property`` resolving
+    possible duplicate testcase elements in ``twister_report.xml`` testsuite container.
+
 Drivers and Sensors
 *******************
 

scripts/pylib/twister/twisterlib/environment.py

@@ -148,12 +148,10 @@ def add_parse_arguments(parser = None) -> argparse.ArgumentParser:
 
     test_plan_report_xor.add_argument("--list-tests", action="store_true",
                              help="""List of all sub-test functions recursively found in
-        all --testsuite-root arguments. Note different sub-tests can share
-        the same test scenario identifier (section.subsection)
-        and come from different directories.
-        The output is flattened and reports --sub-test names only,
-        not their directories. For instance net.socket.getaddrinfo_ok
-        and net.socket.fd_set belong to different directories.
+        all --testsuite-root arguments. The output is flattened and reports detailed
+        sub-test names without their directories.
+        Note: sub-test names can share the same test scenario identifier prefix
+        (section.subsection) even if they are from different test projects.
         """)
 
     test_plan_report_xor.add_argument("--test-tree", action="store_true",
@@ -264,9 +262,11 @@ def add_parse_arguments(parser = None) -> argparse.ArgumentParser:
         functions. Sub-tests are named by:
         'section.subsection_in_testcase_yaml.ztest_suite.ztest_without_test_prefix'.
         Example_1: 'kernel.fifo.fifo_api_1cpu.fifo_loop' where 'kernel.fifo' is a test scenario
-        name (section.subsection) and 'fifo_api_1cpu.fifo_loop' is
-        a Ztest suite_name.test_name identificator.
+        name (section.subsection) and 'fifo_api_1cpu.fifo_loop' is a Ztest 'suite_name.test_name'.
         Example_2: 'debug.coredump.logging_backend' is a standalone test scenario name.
+        Note: This selection mechanism works only for Ztest suite and test function names in
+        the source files which are not generated by macro-substitutions.
+        Note: With --no-detailed-test-id use only Ztest names without scenario name.
         """)
 
     parser.add_argument(
@@ -578,15 +578,21 @@ def add_parse_arguments(parser = None) -> argparse.ArgumentParser:
 
     parser.add_argument(
         '--detailed-test-id', action='store_true',
-        help="Include paths to tests' locations in tests' names. Names will follow "
-             "PATH_TO_TEST/SCENARIO_NAME schema "
-             "e.g. samples/hello_world/sample.basic.helloworld")
+        help="Compose each test Suite name from its configuration path (relative to root) and "
+             "the appropriate Scenario name using PATH_TO_TEST_CONFIG/SCENARIO_NAME schema. "
+             "Also (for Ztest only), prefix each test Case name with its Scenario name. "
+             "For example: 'kernel.common.timing' Scenario with test Suite name "
+             "'tests/kernel/sleep/kernel.common.timing' and 'kernel.common.timing.sleep.usleep' "
+             "test Case (where 'sleep' is its Ztest suite name and 'usleep' is Ztest test name.")
 
     parser.add_argument(
         "--no-detailed-test-id", dest='detailed_test_id', action="store_false",
-        help="Don't put paths into tests' names. "
-             "With this arg a test name will be a scenario name "
-             "e.g. sample.basic.helloworld.")
+        help="Don't prefix each test Suite name with its configuration path, "
+             "so it is the same as the appropriate Scenario name. "
+             "Also (for Ztest only), don't prefix each Ztest Case name with its Scenario name. "
+             "For example: 'kernel.common.timing' Scenario name, the same Suite name, "
+             "and 'sleep.usleep' test Case (where 'sleep' is its Ztest suite name "
+             "and 'usleep' is Ztest test name.")
 
     # Include paths in names by default.
     parser.set_defaults(detailed_test_id=True)

scripts/pylib/twister/twisterlib/harness.py

@@ -767,7 +767,7 @@ def get_testcase(self, tc_name, phase, ts_name=None):
         for ts_name_ in ts_names:
             if self.started_suites[ts_name_]['count'] < (0 if phase == 'TS_SUM' else 1):
                 continue
-            tc_fq_id = f"{self.id}.{ts_name_}.{tc_name}"
+            tc_fq_id = self.instance.compose_case_name(f"{ts_name_}.{tc_name}")
             if tc := self.instance.get_case_by_name(tc_fq_id):
                 if self.trace:
                     logger.debug(f"On {phase}: Ztest case '{tc_name}' matched to '{tc_fq_id}")
@@ -776,7 +776,7 @@ def get_testcase(self, tc_name, phase, ts_name=None):
             f"On {phase}: Ztest case '{tc_name}' is not known"
             f" in {self.started_suites} running suite(s)."
         )
-        tc_id = f"{self.id}.{tc_name}"
+        tc_id = self.instance.compose_case_name(tc_name)
         return self.instance.get_case_or_create(tc_id)
 
     def start_suite(self, suite_name):

scripts/pylib/twister/twisterlib/reports.py

@@ -171,14 +171,14 @@ def xunit_report_suites(self, json_file, filename):
             runnable = suite.get('runnable', 0)
             duration += float(handler_time)
             ts_status = TwisterStatus(suite.get('status'))
+            classname = PosixPath(suite.get("name","")).name
             for tc in suite.get("testcases", []):
                 status = TwisterStatus(tc.get('status'))
                 reason = tc.get('reason', suite.get('reason', 'Unknown'))
                 log = tc.get("log", suite.get("log"))
 
                 tc_duration = tc.get('execution_time', handler_time)
                 name = tc.get("identifier")
-                classname = ".".join(name.split(".")[:2])
                 fails, passes, errors, skips = self.xunit_testcase(eleTestsuite,
                     name, classname, status, ts_status, reason, tc_duration, runnable,
                     (fails, passes, errors, skips), log, True)
@@ -191,6 +191,7 @@ def xunit_report_suites(self, json_file, filename):
             eleTestsuite.attrib['skipped'] = f"{skips}"
             eleTestsuite.attrib['tests'] = f"{total}"
 
+        ET.indent(eleTestsuites, space="\t", level=0)
         result = ET.tostring(eleTestsuites)
         with open(filename, 'wb') as report:
             report.write(result)
@@ -252,14 +253,14 @@ def xunit_report(self, json_file, filename, selected_platform=None, full_report=
                 ):
                     continue
                 if full_report:
+                    classname = PosixPath(ts.get("name","")).name
                     for tc in ts.get("testcases", []):
                         status = TwisterStatus(tc.get('status'))
                         reason = tc.get('reason', ts.get('reason', 'Unknown'))
                         log = tc.get("log", ts.get("log"))
 
                         tc_duration = tc.get('execution_time', handler_time)
                         name = tc.get("identifier")
-                        classname = ".".join(name.split(".")[:2])
                         fails, passes, errors, skips = self.xunit_testcase(eleTestsuite,
                             name, classname, status, ts_status, reason, tc_duration, runnable,
                             (fails, passes, errors, skips), log, True)
@@ -280,6 +281,7 @@ def xunit_report(self, json_file, filename, selected_platform=None, full_report=
             eleTestsuite.attrib['skipped'] = f"{skips}"
             eleTestsuite.attrib['tests'] = f"{total}"
 
+        ET.indent(eleTestsuites, space="\t", level=0)
         result = ET.tostring(eleTestsuites)
         with open(filename, 'wb') as report:
             report.write(result)

scripts/pylib/twister/twisterlib/runner.py

@@ -1186,12 +1186,8 @@ def demangle(self, symbol_name):
         return symbol_name
 
     def determine_testcases(self, results):
-        yaml_testsuite_name = self.instance.testsuite.id
-        logger.debug(f"Determine test cases for test suite: {yaml_testsuite_name}")
+        logger.debug(f"Determine test cases for test suite: {self.instance.testsuite.id}")
 
-        logger.debug(
-            f"Test instance {self.instance.name} already has {len(self.instance.testcases)} cases."
-        )
         new_ztest_unit_test_regex = re.compile(r"z_ztest_unit_test__([^\s]+?)__([^\s]*)")
         detected_cases = []
 
@@ -1220,9 +1216,14 @@ def determine_testcases(self, results):
                                 f"not present in: {self.instance.testsuite.ztest_suite_names}"
                             )
                         test_func_name = m_[2].replace("test_", "", 1)
-                        testcase_id = f"{yaml_testsuite_name}.{new_ztest_suite}.{test_func_name}"
+                        testcase_id = self.instance.compose_case_name(
+                            f"{new_ztest_suite}.{test_func_name}"
+                        )
                         detected_cases.append(testcase_id)
 
+        logger.debug(
+            f"Test instance {self.instance.name} already has {len(self.instance.testcases)} cases."
+        )
         if detected_cases:
             logger.debug(f"Detected Ztest cases: [{', '.join(detected_cases)}] in {elf_file}")
             tc_keeper = {
@@ -1232,16 +1233,17 @@ def determine_testcases(self, results):
             self.instance.testcases.clear()
             self.instance.testsuite.testcases.clear()
 
-            # When the old regex-based test case collection is fully deprecated,
-            # this will be the sole place where test cases get added to the test instance.
-            # Then we can further include the new_ztest_suite info in the testcase_id.
-
             for testcase_id in detected_cases:
                 testcase = self.instance.add_testcase(name=testcase_id)
                 self.instance.testsuite.add_testcase(name=testcase_id)
 
                 # Keep previous statuses and reasons
                 tc_info = tc_keeper.get(testcase_id, {})
+                if not tc_info and self.trace:
+                    # Also happens when Ztest uses macroses, eg. DEFINE_TEST_VARIANT
+                    logger.debug(f"Ztest case '{testcase_id}' discovered for "
+                                 f"'{self.instance.testsuite.source_dir_rel}' "
+                                 f"with {list(tc_keeper)}")
                 testcase.status = tc_info.get('status', TwisterStatus.NONE)
                 testcase.reason = tc_info.get('reason')
 

scripts/pylib/twister/twisterlib/testinstance.py

@@ -1,6 +1,6 @@
 # vim: set syntax=python ts=4 :
 #
-# Copyright (c) 2018-2022 Intel Corporation
+# Copyright (c) 2018-2024 Intel Corporation
 # Copyright 2022 NXP
 # Copyright (c) 2024 Arm Limited (or its affiliates). All rights reserved.
 #
@@ -173,6 +173,9 @@ def __setstate__(self, d):
     def __lt__(self, other):
         return self.name < other.name
 
+    def compose_case_name(self, tc_name) -> str:
+        return self.testsuite.compose_case_name(tc_name)
+
     def set_case_status_by_name(self, name, status, reason=None):
         tc = self.get_case_or_create(name)
         tc.status = status

scripts/pylib/twister/twisterlib/testplan.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 # vim: set syntax=python ts=4 :
 #
-# Copyright (c) 2018 Intel Corporation
+# Copyright (c) 2018-2024 Intel Corporation
 # Copyright (c) 2024 Arm Limited (or its affiliates). All rights reserved.
 #
 # SPDX-License-Identifier: Apache-2.0
@@ -346,9 +346,13 @@ def handle_modules(self):
 
     def report(self):
         if self.options.test_tree:
+            if not self.options.detailed_test_id:
+                logger.info("Test tree is always shown with detailed test-id.")
             self.report_test_tree()
             return 0
         elif self.options.list_tests:
+            if not self.options.detailed_test_id:
+                logger.info("Test list is always shown with detailed test-id.")
             self.report_test_list()
             return 0
         elif self.options.list_tags:
@@ -551,18 +555,18 @@ def get_tests_list(self):
             for _, ts in self.testsuites.items():
                 if ts.tags.intersection(tag_filter):
                     for case in ts.testcases:
-                        testcases.append(case.name)
+                        testcases.append(case.detailed_name)
         else:
             for _, ts in self.testsuites.items():
                 for case in ts.testcases:
-                    testcases.append(case.name)
+                    testcases.append(case.detailed_name)
 
         if exclude_tag := self.options.exclude_tag:
             for _, ts in self.testsuites.items():
                 if ts.tags.intersection(exclude_tag):
                     for case in ts.testcases:
-                        if case.name in testcases:
-                            testcases.remove(case.name)
+                        if case.detailed_name in testcases:
+                            testcases.remove(case.detailed_name)
         return testcases
 
     def add_testsuites(self, testsuite_filter=None):

scripts/pylib/twister/twisterlib/testsuite.py

@@ -386,6 +386,10 @@ def __init__(self, name=None, testsuite=None):
         self.output = ""
         self.freeform = False
 
+    @property
+    def detailed_name(self) -> str:
+        return TestSuite.get_case_name_(self.testsuite, self.name, detailed=True)
+
     @property
     def status(self) -> TwisterStatus:
         return self._status
@@ -477,20 +481,31 @@ def load(self, data):
                 'Harness config error: console harness defined without a configuration.'
             )
 
+    @staticmethod
+    def get_case_name_(test_suite, tc_name, detailed=True) -> str:
+        return f"{test_suite.id}.{tc_name}" \
+            if test_suite and detailed and not test_suite.detailed_test_id else f"{tc_name}"
+
+    @staticmethod
+    def compose_case_name_(test_suite, tc_name) -> str:
+        return f"{test_suite.id}.{tc_name}" \
+            if test_suite and test_suite.detailed_test_id else f"{tc_name}"
+
+    def compose_case_name(self, tc_name) -> str:
+        return self.compose_case_name_(self, tc_name)
+
     def add_subcases(self, data, parsed_subcases=None, suite_names=None):
         testcases = data.get("testcases", [])
         if testcases:
             for tc in testcases:
-                self.add_testcase(name=f"{self.id}.{tc}")
+                self.add_testcase(name=self.compose_case_name(tc))
         else:
             if not parsed_subcases:
                 self.add_testcase(self.id, freeform=True)
             else:
                 # only add each testcase once
-                for sub in set(parsed_subcases):
-                    name = f"{self.id}.{sub}"
-                    self.add_testcase(name)
-
+                for tc in set(parsed_subcases):
+                    self.add_testcase(name=self.compose_case_name(tc))
         if suite_names:
             self.ztest_suite_names = suite_names