twister: ztest: short test case names on --no-detailed-test-id #82302

golowanow · 2024-11-28T22:22:29Z

Extend --no-detailed-test-id command line option: in addition to its current behavior to exclude from a test Suite name its configuration path prefix, also don't prefix each Ztest Case name with its Scenario name.

For example:
kernel.common.timing Scenario name is the same as Suite name, and sleep.usleep test Case where sleep is its Ztest suite name and usleep is Ztest test name.
With detailed-id: tests/kernel/sleep/kernel.common.timing and kernel.common.timing.sleep.usleep respectively.

This way both TestSuite and TestCase names follow the same principle having no parent object name prefix.

There is no information loss in Twister reports with this naming: TestSuite is a container object for its TestCases, whereas TestSuite has its configuration path as a property.

Follow-up to review #80088 and recent discussions at the Twster WG meetings.

golowanow · 2024-11-29T17:01:21Z

rebased resolving conflicts with #82043

nashif · 2024-12-04T18:42:35Z

scripts/pylib/twister/twisterlib/environment.py

That note at the end i do not understand, what is Ztest Identificator? sounds like a new terminology.

I would also reduce verbosity in the help for command line option and prefer to expand in the documentation but this seems to be a common theme across all options, need to clean this up a bit, we have too many options and each comes with a mouthful of help making --help less helpful :(

Anyways, what I noticed is the following:

./scripts/twister --test-tree -T tests/kernel/semaphore/ --no-detailed-test-id ZEPHYR_BASE unset, using "/home/nashif/zephyrproject/zephyr" Renaming output directory to /home/nashif/zephyrproject/zephyr/twister-out.8 INFO - Using Ninja.. INFO - Zephyr version: v4.0.0-1232-gcacd64a67ae1 INFO - Using 'zephyr' toolchain. Testsuite ├── Samples └── Tests ├── kernel_sys_sem ├── semaphore ├── semaphore_1cpu └── semaphore_null_case

vs

./scripts/twister --test-tree -T tests/kernel/semaphore/ ZEPHYR_BASE unset, using "/home/nashif/zephyrproject/zephyr" Renaming output directory to /home/nashif/zephyrproject/zephyr/twister-out.9 INFO - Using Ninja.. INFO - Zephyr version: v4.0.0-1232-gcacd64a67ae1 INFO - Using 'zephyr' toolchain. Testsuite ├── Samples └── Tests └── kernel └── semaphore ├── kernel.semaphore.semaphore.k_sem_correct_count_limit ├── kernel.semaphore.semaphore.k_sem_define ├── kernel.semaphore.semaphore.k_sem_init ├── kernel.semaphore.semaphore.sem_count_get ├── kernel.semaphore.semaphore.sem_give_from_isr ├── kernel.semaphore.semaphore.sem_give_from_thread ├── kernel.semaphore.semaphore.sem_give_take_from_isr ├── kernel.semaphore.semaphore.sem_measure_timeout_from_thread ├── kernel.semaphore.semaphore.sem_measure_timeouts ├── kernel.semaphore.semaphore.sem_multi_take_timeout_diff_sem ├── kernel.semaphore.semaphore.sem_multiple_threads_wait ├── kernel.semaphore.semaphore.sem_reset ├── kernel.semaphore.semaphore.sem_reset_waiting ├── kernel.semaphore.semaphore.sem_take_multiple ├── kernel.semaphore.semaphore.sem_take_no_wait ├── kernel.semaphore.semaphore.sem_take_no_wait_fails ├── kernel.semaphore.semaphore.sem_take_timeout ├── kernel.semaphore.semaphore.sem_take_timeout_fails ├── kernel.semaphore.semaphore.sem_take_timeout_forever ├── kernel.semaphore.semaphore.sem_thread2isr ├── kernel.semaphore.semaphore.sem_thread2thread ├── kernel.semaphore.semaphore_1cpu.sem_multiple_take_and_timeouts ├── kernel.semaphore.semaphore_1cpu.sem_queue_mutual_exclusion ├── kernel.semaphore.semaphore_1cpu.sem_take_timeout_isr ├── kernel.semaphore.semaphore_null_case.sem_count_get_null ├── kernel.semaphore.semaphore_null_case.sem_give_null ├── kernel.semaphore.semaphore_null_case.sem_init_null ├── kernel.semaphore.semaphore_null_case.sem_reset_null ├── kernel.semaphore.semaphore_null_case.sem_take_null ├── kernel.semaphore.usage.kernel_sys_sem.multi_thread_sem_limit └── kernel.semaphore.usage.kernel_sys_sem.multiple_thread_sem_usage

So, this is completeltly different output.

That note at the end i do not understand, what is Ztest Identificator? sounds like a new terminology.

changed

Anyways, what I noticed is the following:
... So, this is completeltly different output.

fixed with Twister tests extended.

Extend `--no-detailed-test-id` command line option: in addition to its current behavior to exclude from a test Suite name its configuration path prefix, also don't prefix each Ztest Case name with its Scenario name. For example: 'kernel.common.timing' Scenario name, the same Suite name, and 'sleep.usleep' test Case (where 'sleep' is its Ztest suite name and 'usleep' is Ztest test name. This way both TestSuite and TestCase names follow the same principle having no parent object name prefix. There is no information loss in Twister reports with this naming: TestSuite is a container object for its TestCases, whereas TestSuite has its configuration path as a property. Signed-off-by: Dmitrii Golovanov <[email protected]>

nashif · 2024-12-15T13:39:50Z

Look at the output in twister-out/twister_suite_report.xml. Right now:

<testsuites>
  <testsuite name="tests/kernel/semaphore/sys_sem/kernel.semaphore.usage" time="1.12" timestamp="2024-12-15T08:36:49.347315" tests="2" failures="0" errors="0" skipped="0">
    <properties>
      <property name="version" value="v4.0.0-2078-geec798390755"/>
      <property name="platform" value="qemu_x86/atom"/>
      <property name="architecture" value="x86"/>
    </properties>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.usage.kernel_sys_sem.multi_thread_sem_limit" time="0.12"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.usage.kernel_sys_sem.multiple_thread_sem_usage" time="0.09"/>
  </testsuite>
  <testsuite name="tests/kernel/semaphore/semaphore/kernel.semaphore" time="1.28" timestamp="2024-12-15T08:36:49.347315" tests="29" failures="0" errors="0" skipped="0">
    <properties>
      <property name="version" value="v4.0.0-2078-geec798390755"/>
      <property name="platform" value="qemu_x86/atom"/>
      <property name="architecture" value="x86"/>
    </properties>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore_null_case.sem_count_get_null" time="0.01"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore_null_case.sem_reset_null" time="0.01"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore_null_case.sem_take_null" time="0.01"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore_null_case.sem_init_null" time="0.01"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore_null_case.sem_give_null" time="0.01"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore_1cpu.sem_queue_mutual_exclusion" time="0.18"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.sem_multi_take_timeout_diff_sem" time="5.01"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore_1cpu.sem_multiple_take_and_timeouts" time="1.25"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.sem_measure_timeout_from_thread" time="0.00"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.sem_measure_timeouts" time="1.01"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.sem_multiple_threads_wait" time="2.04"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.sem_give_take_from_isr" time="0.00"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.k_sem_correct_count_limit" time="0.00"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.sem_take_multiple" time="1.01"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore_1cpu.sem_take_timeout_isr" time="0.00"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.sem_take_timeout_forever" time="0.11"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.sem_take_timeout" time="0.00"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.sem_take_timeout_fails" time="0.54"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.sem_take_no_wait_fails" time="0.00"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.sem_take_no_wait" time="0.00"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.sem_give_from_thread" time="0.00"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.sem_give_from_isr" time="0.00"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.sem_count_get" time="0.00"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.sem_reset_waiting" time="0.02"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.sem_reset" time="0.11"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.k_sem_init" time="0.00"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.sem_thread2isr" time="0.00"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.sem_thread2thread" time="0.00"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.semaphore.k_sem_define" time="0.00"/>
  </testsuite>
</testsuites>

with this PR:

<testsuites>
  <testsuite name="kernel.semaphore.usage" time="1.22" timestamp="2024-12-15T08:33:34.439613" tests="2" failures="0" errors="0" skipped="0">
    <properties>
      <property name="version" value="v4.0.0-2078-geec798390755"/>
      <property name="platform" value="qemu_x86/atom"/>
      <property name="architecture" value="x86"/>
    </properties>
    <testcase classname="kernel_sys_sem.multi_thread_sem_limit" name="kernel_sys_sem.multi_thread_sem_limit" time="0.12"/>
    <testcase classname="kernel_sys_sem.multiple_thread_sem_usage" name="kernel_sys_sem.multiple_thread_sem_usage" time="0.09"/>
  </testsuite>
  <testsuite name="kernel.semaphore" time="1.15" timestamp="2024-12-15T08:33:34.439613" tests="29" failures="0" errors="0" skipped="0">
    <properties>
      <property name="version" value="v4.0.0-2078-geec798390755"/>
      <property name="platform" value="qemu_x86/atom"/>
      <property name="architecture" value="x86"/>
    </properties>
    <testcase classname="semaphore_null_case.sem_count_get_null" name="semaphore_null_case.sem_count_get_null" time="0.01"/>
    <testcase classname="semaphore_null_case.sem_reset_null" name="semaphore_null_case.sem_reset_null" time="0.01"/>
    <testcase classname="semaphore_null_case.sem_take_null" name="semaphore_null_case.sem_take_null" time="0.01"/>
    <testcase classname="semaphore_null_case.sem_init_null" name="semaphore_null_case.sem_init_null" time="0.01"/>
    <testcase classname="semaphore_null_case.sem_give_null" name="semaphore_null_case.sem_give_null" time="0.01"/>
    <testcase classname="semaphore_1cpu.sem_queue_mutual_exclusion" name="semaphore_1cpu.sem_queue_mutual_exclusion" time="0.18"/>
    <testcase classname="semaphore.sem_multi_take_timeout_diff_sem" name="semaphore.sem_multi_take_timeout_diff_sem" time="5.01"/>
    <testcase classname="semaphore_1cpu.sem_multiple_take_and_timeouts" name="semaphore_1cpu.sem_multiple_take_and_timeouts" time="1.25"/>
    <testcase classname="semaphore.sem_measure_timeout_from_thread" name="semaphore.sem_measure_timeout_from_thread" time="0.00"/>
    <testcase classname="semaphore.sem_measure_timeouts" name="semaphore.sem_measure_timeouts" time="1.01"/>
    <testcase classname="semaphore.sem_multiple_threads_wait" name="semaphore.sem_multiple_threads_wait" time="2.04"/>
    <testcase classname="semaphore.sem_give_take_from_isr" name="semaphore.sem_give_take_from_isr" time="0.00"/>
    <testcase classname="semaphore.k_sem_correct_count_limit" name="semaphore.k_sem_correct_count_limit" time="0.00"/>
    <testcase classname="semaphore.sem_take_multiple" name="semaphore.sem_take_multiple" time="1.01"/>
    <testcase classname="semaphore_1cpu.sem_take_timeout_isr" name="semaphore_1cpu.sem_take_timeout_isr" time="0.00"/>
    <testcase classname="semaphore.sem_take_timeout_forever" name="semaphore.sem_take_timeout_forever" time="0.11"/>
    <testcase classname="semaphore.sem_take_timeout" name="semaphore.sem_take_timeout" time="0.00"/>
    <testcase classname="semaphore.sem_take_timeout_fails" name="semaphore.sem_take_timeout_fails" time="0.54"/>
    <testcase classname="semaphore.sem_take_no_wait_fails" name="semaphore.sem_take_no_wait_fails" time="0.00"/>
    <testcase classname="semaphore.sem_take_no_wait" name="semaphore.sem_take_no_wait" time="0.00"/>
    <testcase classname="semaphore.sem_give_from_thread" name="semaphore.sem_give_from_thread" time="0.00"/>
    <testcase classname="semaphore.sem_give_from_isr" name="semaphore.sem_give_from_isr" time="0.00"/>
    <testcase classname="semaphore.sem_count_get" name="semaphore.sem_count_get" time="0.00"/>
    <testcase classname="semaphore.sem_reset_waiting" name="semaphore.sem_reset_waiting" time="0.02"/>
    <testcase classname="semaphore.sem_reset" name="semaphore.sem_reset" time="0.11"/>
    <testcase classname="semaphore.k_sem_init" name="semaphore.k_sem_init" time="0.00"/>
    <testcase classname="semaphore.sem_thread2isr" name="semaphore.sem_thread2isr" time="0.00"/>
    <testcase classname="semaphore.sem_thread2thread" name="semaphore.sem_thread2thread" time="0.00"/>
    <testcase classname="semaphore.k_sem_define" name="semaphore.k_sem_define" time="0.00"/>
  </testsuite>
</testsuites>

classname should be kernel.semaphore as before, with testcase excluding the classname, so

<testcase classname="kernel.semaphore" name="semaphore.k_sem_define" time="0.00"/>

The testcase.classname properties in `twister_report.xml` and `twister_suite_report.xml` now contain the full testsuite name (without its path prefix) instead of just two its dot-separated leftmost components. This way `twister_report.xml` testsuite container has no duplicate testcase elements with the same `classname` and `name` properties executing with --no-detailed-test-id for a project with same testcase names in its 'sibling' suites, for example`tests/kernel/sleep`. Signed-off-by: Dmitrii Golovanov <[email protected]>

Pretty print XML reports with tab identation on levels for better readability and ease text diff there as well. Signed-off-by: Dmitrii Golovanov <[email protected]>

golowanow · 2024-12-15T17:47:38Z

classname should be kernel.semaphore as before, with testcase excluding the classname, so
<testcase classname="kernel.semaphore" name="semaphore.k_sem_define" time="0.00"/>

fixed, also addressing duplicate testcase elements possible in twister_report.xml with --no-detailed-test-id mode and shortened case names in 'sibling' suites (from the same project).

golowanow force-pushed the twister_testcase_shortid_20241127 branch from b51d0fd to d1c841b Compare November 28, 2024 23:37

golowanow added Enhancement Changes/Updates/Additions to existing features area: Twister Twister labels Nov 28, 2024

golowanow force-pushed the twister_testcase_shortid_20241127 branch from d1c841b to 8223e37 Compare November 29, 2024 11:29

golowanow marked this pull request as ready for review November 29, 2024 11:33

zephyrbot added the area: Testsuite Testsuite label Nov 29, 2024

zephyrbot requested review from KamilxPaszkiet, LukaszMrugala, PerMac, aaronemassey, asemjonovs, gchwier, hakehuang, jeremybettis, nashif and yperess November 29, 2024 11:34

zephyrbot assigned nashif Nov 29, 2024

golowanow mentioned this pull request Nov 29, 2024

twister: ztest: harness: Fix missed TestCase statuses #80088

Merged

golowanow force-pushed the twister_testcase_shortid_20241127 branch from 8223e37 to 0df70f9 Compare November 29, 2024 16:57

golowanow force-pushed the twister_testcase_shortid_20241127 branch from 0df70f9 to cacd64a Compare November 29, 2024 17:04

hakehuang previously approved these changes Nov 30, 2024

View reviewed changes

nashif requested changes Dec 4, 2024

View reviewed changes

golowanow dismissed hakehuang’s stale review via 4b8e4da December 10, 2024 20:38

golowanow force-pushed the twister_testcase_shortid_20241127 branch from cacd64a to 4b8e4da Compare December 10, 2024 20:38

zephyrbot added the Release Notes To be mentioned in the release notes label Dec 10, 2024

zephyrbot requested review from fabiobaltieri and kartben December 10, 2024 20:39

golowanow requested review from hakehuang and nashif December 10, 2024 20:44

golowanow force-pushed the twister_testcase_shortid_20241127 branch from 4b8e4da to ca73979 Compare December 10, 2024 20:49

golowanow force-pushed the twister_testcase_shortid_20241127 branch from ca73979 to a0b13a6 Compare December 10, 2024 21:50

golowanow added 2 commits December 15, 2024 18:34

twister: reports: Pretty print XML reports

0bdfc99

Pretty print XML reports with tab identation on levels for better readability and ease text diff there as well. Signed-off-by: Dmitrii Golovanov <[email protected]>

nashif approved these changes Dec 15, 2024

View reviewed changes

hakehuang approved these changes Dec 16, 2024

View reviewed changes

kartben merged commit 2c5195c into zephyrproject-rtos:main Dec 16, 2024
32 checks passed

kartben mentioned this pull request Dec 16, 2024

twister: reports: use Path instead of PosixPath #83030

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

twister: ztest: short test case names on --no-detailed-test-id #82302

twister: ztest: short test case names on --no-detailed-test-id #82302

Uh oh!

golowanow commented Nov 28, 2024 •

edited

Loading

Uh oh!

golowanow commented Nov 29, 2024

Uh oh!

nashif Dec 4, 2024

Uh oh!

golowanow Dec 10, 2024

Uh oh!

nashif commented Dec 15, 2024

Uh oh!

golowanow commented Dec 15, 2024 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

twister: ztest: short test case names on --no-detailed-test-id #82302

twister: ztest: short test case names on --no-detailed-test-id #82302

Uh oh!

Conversation

golowanow commented Nov 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

golowanow commented Nov 29, 2024

Uh oh!

nashif Dec 4, 2024

Choose a reason for hiding this comment

Uh oh!

golowanow Dec 10, 2024

Choose a reason for hiding this comment

Uh oh!

nashif commented Dec 15, 2024

Uh oh!

golowanow commented Dec 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

golowanow commented Nov 28, 2024 •

edited

Loading

golowanow commented Dec 15, 2024 •

edited

Loading