twister: `WARNING - TS_START: already STARTED` on expected test case reboot

[JordanYates]

Is your enhancement proposal related to a problem? Please describe.

When using twister with testcases that are expected to reboot (i.e. testing fault handling), numerous warning are output to the terminal.

WARNING - TS_START: already STARTED 'infuse_reboot':{'count': 1, 'repeat': 0}
WARNING - TC_START: already STARTED case 'lib.reboot.a.infuse_reboot.reboot':{'count': 1}
WARNING - TS_START: already STARTED 'infuse_reboot':{'count': 2, 'repeat': 1}
WARNING - TC_START: already STARTED case 'lib.reboot.a.infuse_reboot.reboot':{'count': 2}
WARNING - TS_START: already STARTED 'infuse_reboot':{'count': 3, 'repeat': 2}
WARNING - TC_START: already STARTED case 'lib.reboot.a.infuse_reboot.reboot':{'count': 3}
WARNING - TS_START: already STARTED 'infuse_reboot':{'count': 1, 'repeat': 0}
WARNING - TC_START: already STARTED case 'lib.reboot.b.infuse_reboot.reboot':{'count': 1}
WARNING - TS_START: already STARTED 'infuse_reboot':{'count': 2, 'repeat': 1}
WARNING - TC_START: already STARTED case 'lib.reboot.b.infuse_reboot.reboot':{'count': 2}
WARNING - TS_START: already STARTED 'infuse_reboot':{'count': 3, 'repeat': 2}
WARNING - TC_START: already STARTED case 'lib.reboot.b.infuse_reboot.reboot':{'count': 3}

Reproduction

./scripts/twister -T tests/drivers/flash_simulator/flash_sim_reboot/ --integration
INFO    - Using Ninja..
INFO    - Zephyr version: v4.1.0-rc1-21-g059e86f426b8
INFO    - Using 'zephyr' toolchain.
INFO    - Selecting default platforms per testsuite scenario
INFO    - Building initial testsuite list...
INFO    - Writing JSON report /home/jordan/code/zephyr/zephyr/twister-out/testplan.json
INFO    - JOBS: 28
INFO    - Adding tasks to the queue...
INFO    - Added initial list of jobs to queue
WARNING - TS_START: already STARTED 'flash_sim_reboot':{'count': 1, 'repeat': 0}
WARNING - TC_START: already STARTED case 'drivers.flash.flash_simulator.boot_no_erase.flash_sim_reboot.preserve_over_reboot' {'count': 1}
INFO    - Total complete:    1/   1  100%  built (not run):    0, filtered:    0, failed:    0, error:    0
INFO    - 1 test scenarios (1 configurations) selected, 0 configurations filtered (0 by static filter, 0 at runtime).
INFO    - 1 of 1 executed test configurations passed (100.00%), 0 built (not run), 0 failed, 0 errored, with no warnings in 8.29 seconds.
INFO    - 1 of 1 executed test cases passed (100.00%) on 1 out of total 948 platforms (0.11%).
INFO    - 1 test configurations executed on platforms, 0 test configurations were only built.
INFO    - Saving reports...
INFO    - Writing JSON report /home/jordan/code/zephyr/zephyr/twister-out/twister.json
INFO    - Writing xunit report /home/jordan/code/zephyr/zephyr/twister-out/twister.xml...
INFO    - Writing xunit report /home/jordan/code/zephyr/zephyr/twister-out/twister_report.xml...

Describe the solution you'd like

A way to specify that the test is expected to reboot in order to supress the warnings

Additional context

Messages originally added in #80088.

[JordanYates]

Initial investigation makes it seem like this will be tricky to implement, since the location of the warning has no context at all about the testcase that the test is a part of.

zephyr/scripts/pylib/twister/twisterlib/harness.py

Line 839 in 8a814dd

logger.warning(f"{phase}: already STARTED '{suite_name}':"

[nashif]

@golowanow can you please take a look?

[JordanYates]

My initial feeling is that the easiest way to do this would be to define a ZTEST_REBOOTS macro to use instead of ZTEST.
The macro would save a flag a in the testcase struct, which in turn would be checked at the start of the test to print START_REBOOT - instead of START -, so the python side knows that the reboots are normal.
The testsuite could iterate over the testcases it contains in order to discover if any of its constituents expect to reboot to similarly change the starting message.

[golowanow]

When using twister with testcases that are expected to reboot (i.e. testing fault handling), numerous warning are output to the terminal.

right, these warnings also happen with this particular kind of test cases and there is at least one more known already tests/drivers/watchdog/wdt_basic_api.

My initial feeling is that the easiest way to do this would be to define a ZTEST_REBOOTS macro to use instead of ZTEST.

to me it also looks reasonable to declare such tests at Ztest and let harness act specifically.

@JordanYates would you like to raise a PR for that already ? or I can do it later, perhaps this week.