Twister // Pytest bug & integration status

[Dermiste]

Hi all,

I am trying to run pytest on a sample project, no success so far, here is a summary of the issue:

Describe the bug
No coherent pytest output when running provided pytest sample, either shell or basic

To Reproduce

1. Copy and paste samples\subsys\testsuite\pytest\basic\pytest folder into tests\lib\cbprintf_fp & overwrite testcase.yaml.
2. Modify platform-allow and include nucleo_u575zi_q
3. Run twister with device mode on that last folder:
   west twister --device-testing --device-serial COM3 -p nucleo_u575zi_q -T tests\lib\cbprintf_fp

Please also mention any information which could help others to understand
the problem you're facing:
Windows 11
zepyhr 3.5.0 , the one installed by install script proposed in https://docs.zephyrproject.org/latest/develop/getting_started/index.html#:~:text=Get%20the%20Zephyr%20source%20code%3A

Actual behaviour
Same error pops up happens no matter what I try:

Error when parsing file C:\Users\ABC\zephyrproject\zephyr\twister-out\nucleo_u575zi_q\tests/lib/cbprintf_fp/sample.twister.pytest\report.xml: [Errno 2] No such file or directory: 'C:\\Users\\ABC\\zephyrproject\\zephyr\\twister-out\\nucleo_u575zi_q\\tests/lib/cbprintf_fp/sample.twister.pytest\\report.xml'
see: ←[33mC:\Users\ABC\zephyrproject\zephyr\twister-out\nucleo_u575zi_q\tests\lib\cbprintf_fp\sample.twister.pytest\build.log←[39m

Expected behavior
Some actual pytest results

Impact
Impossibility to try Zephyr Twister Pytest features

Full consoleoutput

C:\Users\ABC\zephyrproject\zephyr>west twister --device-testing --device-serial COM3 -p nucleo_u575zi_q -T tests\lib\cbprintf_fp
Renaming output directory to C:\Users\ABC\zephyrproject\zephyr\twister-out.46
INFO    - Using Ninja..
INFO    - Zephyr version: zephyr-v3.5.0-2054-geeb6bf7dd998
INFO    - Using 'zephyr' toolchain.
INFO    - Building initial testsuite list...
INFO    - Writing JSON report C:\Users\ABC\zephyrproject\zephyr\twister-out\testplan.json

Device testing on:

| Platform        | ID   | Serial device   |
|-----------------|------|-----------------|
| nucleo_u575zi_q |      | COM3            |

INFO    - Adding tasks to the queue...
INFO    - Added initial list of jobs to queue
Error when parsing file C:\Users\ABC\zephyrproject\zephyr\twister-out\nucleo_u575zi_q\tests/lib/cbprintf_fp/sample.twister.pytest\report.xml: [Errno 2] No such file or directory: 'C:\\Users\\ABC\\zephyrproject\\zephyr\\twister-out\\nucleo_u575zi_q\\tests/lib/cbprintf_fp/sample.twister.pytest\\report.xml'
nucleo_u575zi_q           tests/lib/cbprintf_fp/sample.twister.pytest        ←[31m FAILED ←[39m: Unknown
see: ←[33mC:\Users\ABC\zephyrproject\zephyr\twister-out\nucleo_u575zi_q\tests\lib\cbprintf_fp\sample.twister.pytest\build.log←[39m
INFO    - Total complete: ←[32m   1/   1←[39m  100%  skipped: ←[39m   0←[39m, failed: ←[31m   1←[39m, error: ←[39m   0←[39m
INFO    - 1 test scenarios (1 test instances) selected, 0 configurations skipped (0 by static filter, 0 at runtime).
INFO    - ←[31m0 of 1←[39m test configurations passed (0.00%), ←[31m1←[39m failed, ←[39m0←[39m errored, 0 skipped with ←[39m0←[39m warnings in 26.14 seconds
INFO    - In total 1 test cases were executed, 0 skipped on 1 out of total 647 platforms (0.15%)
INFO    - ←[32m0←[39m test configurations executed on platforms, ←[31m1←[39m test configurations were only built.

Hardware distribution summary:

| Board           | ID   |   Counter |
|-----------------|------|-----------|
| nucleo_u575zi_q |      |         1 |
INFO    - Saving reports...
INFO    - Writing JSON report C:\Users\ABC\zephyrproject\zephyr\twister-out\twister.json
INFO    - Writing xunit report C:\Users\ABC\zephyrproject\zephyr\twister-out\twister.xml...
INFO    - Writing xunit report C:\Users\ABC\zephyrproject\zephyr\twister-out\twister_report.xml...
INFO    - -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
INFO    - The following issues were found (showing the top 10 items):
INFO    - 1) tests/lib/cbprintf_fp/sample.twister.pytest on nucleo_u575zi_q failed (Unknown)
INFO    -
INFO    - To rerun the tests, call twister using the following commandline:
INFO    - west twister -p <PLATFORM> -s <TEST ID>, for example:
INFO    -
INFO    - west twister -p nucleo_u575zi_q -s tests/lib/cbprintf_fp/sample.twister.pytest
INFO    - or with west:
INFO    - west build -p -b nucleo_u575zi_q tests\lib\cbprintf_fp -T sample.twister.pytest
INFO    - -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
INFO    - Run completed

Is it supposed to even work properly ? I found this old PR which was merged already, but mentions plugins I don't seem to have in my zephyr tree #57117

Any help much appreciated