twister: scripts: sharing of built applications

[gchwier]

This feature enables sharing of built applications between test scenarios, allowing tests to access build artifacts from other applications.

from twister.rst:

Required applications must be available in the source tree (specified with -T
and/or -s options). When reusing build directories (e.g., with --no-clean),
Twister can find required applications in the current build directory.

How it works:

• Twister builds the required applications first
• The main test application waits for required applications to complete
• Build directories of required applications are made available to the test harness
• For pytest harness, build directories are passed via --required-build arguments
  and accessible through the required_build_dirs fixture

Limitations: Not supported with --subset or --runtime-artifact-cleanup options.

samples/subsys/testsuite/pytest/shell is extended to show how to use that feature.
To test one can run:
./scripts/twister -c -vv -ll info -T samples/subsys/testsuite/pytest/shell -G

if samples were built, one can run it with --no-clean (required builds will be found in build directory):
./scripts/twister --no-clean -vv -ll debug -T samples/subsys/testsuite/pytest/shell -s sample.pytest.required_app_demo -p native_sim

To run on the hardware (must have hardware map or change the command with platform and serial parameters):
./scripts/twister -vv -ll debug -T samples/subsys/testsuite/pytest/shell -s sample.pytest.required_app_demo --device-testing --hardware-map ~/tmp/hardware_map.yml --west-flash -s sample.harness.shell -s sample.pytest.shell

---

Related to RFC: #62649
Based on that PR: #75291 (but removed no_own_image part and added option to re-use previously built applications from outdir)

[hakehuang]

@JingsaiLu this is the function that you asked for bt testing

[JingsaiLu]

hi @gchwier
this is a great new feature, which may also meet some requirements in #83643

I am trying to understand and validate this solution on my boards,

one question,
west twister -T samples/subsys/testsuite/pytest/shell -s sample.pytest.required_app_demo -p mimxrt1170_evk@B/mimxrt1176/cm7
why nothing to be built ? expect this could trigger all build directly.

add -s sample.harness.shell -s sample.pytest.shell will work,

[gchwier]

I am trying to understand and validate this solution on my boards,

one question, west twister -T samples/subsys/testsuite/pytest/shell -s sample.pytest.required_app_demo -p mimxrt1170_evk@B/mimxrt1176/cm7 why nothing to be built ? expect this could trigger all build directly.

Thanks for testing it!
That command does not works, because as wrote above:

Required applications must be available in the source tree (specified with -T
and/or -s options). When reusing build directories (e.g., with --no-clean),

All test applications must be available in test plan. So you can build everything from that folder (skip -s sample.pytest.required_app_demo) or select every testsuite as you wrote (adding -s sample.harness.shell -s sample.pytest.shell). Required applications are not added automatically.
You can also prebuild everything in one step, and then call only your test with --no-clean option.

[JingsaiLu]

Thanks @gchwier for the detailed explanation : ),
it looks redundant a bit, when we run test sample.pytest.required_app_demo only,
the dependency is already defined in test case yaml, but user still needs to describe it in command line again,
Is it possible to add the required applications automatically, or raise error when no required apps in test plan ?

another question is,
how could we only execute pytest test cases of sample.pytest.required_app_demo ?
with you command, all the pytest cases of required apps seems to be also executed.

[JingsaiLu]

one more question, : )

Have you considered how to flash required application images into different devices in one test ?
would you need to rename test dirs in twister-out/twister_links to test name related ?

--required-build=C:\Users\97014\zephyrproject\zephyr\twister-out\twister_links\test_0' '--required-build=C:\Users\97014\zephyrproject\zephyr\twister-out\twister_links\test_1' -p twister_harness.plugin

[fundakol]

This method should be split for multiple methods to reduce its complexity.

[gchwier]

look on apply_filters, this is a continuation ;)
I do not really want to split this method, as it has a sequential nature, the current implementation keeps all related logic (validating and processing required application) together, making it easier to maintain and debug

[gchwier]

it looks redundant a bit, when we run test sample.pytest.required_app_demo only, the dependency is already defined in test case yaml, but user still needs to describe it in command line again, Is it possible to add the required applications automatically,

I agree, it looks redundant. Within that PR there is not possible to add required applications automatically. The current implementation requires explicit specification of required builds on the command line. I could not find any easy solution how to implement it, without too many changes.

or raise error when no required apps in test plan ?

If required apps is not in test plan, then the current test is filtered out. I do not want to raise an error, to not break current test execution. If you are using integration platform (-G in Twister), then this test will have error status in the report (not skipped)

another question is, how could we only execute pytest test cases of sample.pytest.required_app_demo ? with you command, all the pytest cases of required apps seems to be also executed.

Yes, all tests that you specified in the command line will be executed. For now the logic is simple.
You can create your onw test, where all required applications are build_only: true.

I thought about adding required apps automatically, if are not given from the comamnd line - and then mark them as build_only,
but as I mention before, it is not possible in that PR.

Have you considered how to flash required application images into different devices in one test ?

In the current implementation we only pass the build dirs of required apps to the pytest run - and in pytest scenario implementation you can do what you want with that artefacts.
Currently Twister does not support multi-boards. And not every required build will be used to flash the board (which board? for BLE scenarios you probable want to use another device, but how to reserve it? hardware map should be reused?).

If that PR is merged, we can extend it by adding new functionalities in a separate PR. Some of these enhancements would require refactoring several methods and updating unit tests, so it's better to keep them separate since the current PR is already big enough.

[gchwier]

@fundakol Thank you for reviewing the changes.

[gchwier]

look on apply_filters, this is a continuation ;)
I do not really want to split this method, as it has a sequential nature, the current implementation keeps all related logic (validating and processing required application) together, making it easier to maintain and debug

[sonarqubecloud]

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[hakehuang]

#95056 a device for test approach, please take a look