RFC: twister: pytest: multiple boards with multiple zephyr apps for one test

[JingsaiLu]

Problem description

for some samples / tests, especially for the wireless, one test requires >=2 boards for one test,
such as:

• samples\bluetooth\central_ht could be tested against with samples\bluetooth\peripheral_ht on another harness board
• tests\bluetooth\shell could be tested against with the same app on another board for BLE/ BT Classic / LE audio features, and also for WiFi.
• for the BLE mesh / zigbee and open-thread, which may require 3 boards with 3 different apps for the different roles, such as router/coordinator/end device.

currently, twister pytest framework seems to only consider one DuT device for test.
need some improvement to cover above test scenarios in twister framework,
so that we could catch more issues on real board in CI/CV, like this issue on specific HW, #83551

Proposed change

• add harness apps for different harness devices configurations in sample.yml
• add harness devices info config file
• build and flash different harness apps into multi harness boards
• init all harness devices as resources in fixture, like dut for tests.
• write pytest test cases for samples/tests

Concerns and Unresolved Questions

the initial implementation in #83641, only allows the harness_device_map.yml for one device in dut_map.yml,
need more changes in twister to merge or mapping the device_map configuration file.

[henrikbrixandersen]

for some samples / tests, especially for the wireless, one test requires >=2 boards for one test, ...

This is very relevant for all kind of connectivity tests (networking, CAN, ...).

[PerMac]

Tests with multiple images and devices are a tricky topic we were trying to address already for some time.
There are several issues we encountered that needs to be resolved:

• handling multiple images required by a single test:
  • all builds must happen in twister (no building through pytest)
  • we don't want redundant images. we want a mechanism for sharing images between tests. If there are multiple tests which use the same image (e.g. multiple tests can use the same central with different peripherals) we want the image to be built once and then shared between tests. This might require tweaking with a test scheduler and considering a case where --subset is used (required image didn't end in the same subset)
• handling multiple devices in a single tests:
  • where a required setup can be defined
  • haw to link what's available with what's required
  • how generic and flexible such setups should be?
  • can a device work both as dut and "harness"? e.g. if there are several devices connected, can twister pair them to improve parallelism instead of waiting for those marked as "harness" to be free?

We tried with such approach #75291 The PR got stale due to a lack of proper feedback and we didn't got time to continue driving it, but since this topic is important we are considering going back with it.
I think that before any implementation can be accepted we have to first have a general discussion, e.g. during a testing wg meeting, to collect requirements and align our perspective

[JingsaiLu]

Tests with multiple images and devices are a tricky topic we were trying to address already for some time. There are several issues we encountered that needs to be resolved:

• handling multiple images required by a single test:

  • all builds must happen in twister (no building through pytest)
  • we don't want redundant images. we want a mechanism for sharing images between tests. If there are multiple tests which use the same image (e.g. multiple tests can use the same central with different peripherals) we want the image to be built once and then shared between tests. This might require tweaking with a test scheduler and considering a case where --subset is used (required image didn't end in t he same subset)

• handling multiple devices in a single tests:

  • where a required setup can be defined
  • haw to link what's available with what's required
  • how generic and flexible such setups should be?
  • can a device work both as dut and "harness"? e.g. if there are several devices connected, can twister pair them to improve parallelism instead of waiting for those marked as "harness" to be free?

We tried with such approach #75291 The PR got stale due to a lack of proper feedback and we didn't got time to continue driving it, but since this topic is important we are considering going back with it. I think that before any implementation can be accepted we have to first have a general discussion, e.g. during a testing wg meeting, to collect requirements and align our perspective

Hi @PerMac , thanks a lot for your comments,
hope we could have this feature asap with community collaboration.
as it is the best way (easy and efficient) to write tests and run on HW boards for some features/APIs of zephyr ecosystem stack and apps, like LE Localization, BT classic, and LE audio, and even for all kind of connectivity tests (networking, CAN, ...) as @henrikbrixandersen commented.