Add RTT support to Twister and Pytest

[MarkoSagadin]

Summary

This PR fixes shell RTT interactions via west rtt command and adds RTT support in Twister and pytest-twister-harness.
This enabling device testing via RTT in addition to existing UART and serial pty transports.

Details

I have split this work into 3 separate commits, where each one makes changes to the specific part of the code.

I have tried to capture all the details in the commit messages, so check them out, but an overall view of the PR looks like this:

• First commit fixes west rtt so that is suitable for interacting with the shell subsys. Previously, it was only suitable for presenting logs captured via the RTT, meaning that arrow and navigation keys weren't correctly interpreted. Separate Unix and Windows implementations were needed to resolve this.
• Second commit adds RTT support to the pytest-twister-harness.
• Third commit add RTT support to the Twister itself.

I have been mostly testing this with the nRF7002dk board.

Below are some commands that I used to test this.

Testing instructions - click me

Run single ZBus test

Uses Uart for communication (see this as a control test group, I want to show that RTT support didn't break anything else).

west twister \
    -T tests/subsys/zbus/channel_id \
    -p nrf7002dk/nrf5340/cpuapp \
    --device-testing \
    --device-serial /dev/ttyACM1 \
    --log-level DEBUG \
    -vvv

Uses RTT for communication (-x flags add additional flags to the builds, we use them here to enable RTT as logging backend).

west twister \
    -T tests/subsys/zbus/channel_id \
    -p nrf7002dk/nrf5340/cpuapp \
    --device-testing \
    --device-rtt \
    --log-level DEBUG \
    -x=CONFIG_USE_SEGGER_RTT=y \
    -x=CONFIG_LOG_BACKEND_RTT=y \
    -x=CONFIG_LOG_BACKEND_UART=n \
    -x=CONFIG_LOG_MODE_DEFERRED=y \
    -vvv

Run several ZBus tests

They use console or ztest harnesses.

Uses UART:

west twister \
    -T tests/subsys/zbus \
    -p nrf7002dk/nrf5340/cpuapp \
    --device-testing \
    --device-serial /dev/ttyACM1 \
    --log-level DEBUG \
    -vvv

Uses RTT:

west twister \
    -T tests/subsys/zbus \
    -p nrf7002dk/nrf5340/cpuapp \
    --device-testing \
    --device-rtt \
    --log-level DEBUG \
    -x=CONFIG_USE_SEGGER_RTT=y \
    -x=CONFIG_LOG_BACKEND_RTT=y \
    -x=CONFIG_LOG_BACKEND_UART=n \
    -x=CONFIG_LOG_MODE_DEFERRED=y \
    -vvv

Run test using the Pytest harness

Uses UART:

west twister \
    -T samples/subsys/testsuite/pytest/shell \
    --test sample.pytest.shell \
    -p nrf7002dk/nrf5340/cpuapp \
    --device-testing \
    --device-serial /dev/ttyACM1 \
    --log-level DEBUG \
    -vvv

Uses RTT (I have added a specific build configuration as a part of the PR work, that's why the -x flags are not used here, they are part of sample.pytest.rtt:

west twister \
    -T samples/subsys/testsuite/pytest/shell \
    --test sample.pytest.rtt \
    -p nrf7002dk/nrf5340/cpuapp \
    --device-testing \
    --device-rtt \
    --log-level DEBUG \
    -vvv

Run a single test using hardware map

1. generate hardware map with west twister --generate-hardware-map map.yaml.
2. Open it and delete whole one entry (you will get two if you used nrf7002dk, one if you have gp_md_sp1 (I think))
3. Adjust the platform and serial fields, they won't be correct.

How the map.yaml should look like:

- connected: true
  id: 001050795550
  platform: nrf7002dk/nrf5340/cpuapp
  product: J-Link
  runner: nrfutil
  serial: /dev/ttyACM1

How to use it, UART will be used:

west twister \
    -T tests/subsys/zbus/integration \
    --device-testing \
    --hardware-map map.yaml \
    --log-level DEBUG \
    -vvv

To adapt it for RTT copy the map.yaml into map_rtt.yaml and change some fields, see below example:

- connected: true
  id: 001050795550
  platform: nrf7002dk/nrf5340/cpuapp
  product: J-Link
  runner: nrfutil
  rtt: true

How to use it, RTT will be used:

west twister \
    -T tests/subsys/zbus/integration \
    --device-testing \
    --hardware-map map_rtt.yaml \
    --log-level DEBUG \
    -x=CONFIG_USE_SEGGER_RTT=y \
    -x=CONFIG_LOG_BACKEND_RTT=y \
    -x=CONFIG_LOG_BACKEND_UART=n \
    -x=CONFIG_LOG_MODE_DEFERRED=y \
    -vvv

Use Pytest directly, without Twister

This way is a bit nicer for development, since you can decide if want to rebuild binary or just correct the Python tests and rerun them.

cd ~/workdir/zephyr/samples/subsys/testsuite/pytest/shell
west build -b nrf7002dk/nrf5340/cpuapp . -T sample.pytest.rtt
west flash

# Below are needed for the pytest to work directly
source ~/workdir/zephyr/zephyr-env.sh
export PYTHONPATH=~/workdir/zephyr/scripts/pylib/pytest-twister-harness/src:$PYTHONPATH

pytest -s -v \
    -p twister_harness.plugin \
    --twister-harness \
    --device-type=hardware \
    --device-rtt=True \
    --build-dir build \
    --log-cli-level=DEBUG

To test the Pytest via UART use:

cd ~/workdir/zephyr/samples/subsys/testsuite/pytest/shell
west build -b nrf7002dk/nrf5340/cpuapp
west flash

# Below are needed for the pytest to work directly
source ~/workdir/zephyr/zephyr-env.sh
export PYTHONPATH=~/workdir/zephyr/scripts/pylib/pytest-twister-harness/src:$PYTHONPATH

pytest -s -v \
    -p twister_harness.plugin \
    --twister-harness \
    --device-type=hardware \
    --device-serial=/dev/ttyACM1 \
    --build-dir build \
    --log-cli-level=DEBUG

Running the above commands should demonstrate:

• that RTT works for the pytest, ztest and console harnesses via Twister
• that RTT work when pytest is invoked directly
• that UART functionality wasn't broken
• that hardware map works.

Limitations

During implementation of RTT support I ran into a problem. Currently it is not possible to specify two or more device in the hardware map. That is because Twister will try to execute tests on the DUTs in the parralel, meaning that several west rtt commands need to be able to coexist at once. This would require the west rtt command to provide a way to the caller to control all of it's ports, thus preventing any socket clashes.

---

EDIT

I have fixed all failing tests, I have missed them initially. I am not super confident that I have fixed them in correct way nor if I should write more of them that test the added feature. I would be happy to receive to some guidance in regards to this.

[sonarqubecloud]

Quality Gate passed

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.5% Duplication on New Code

See analysis details on SonarQube Cloud

[tpambor]

probe-rs runner supports west rtt as well as --dev-id

Suggested change

	if rtt_runner in ("jlink", "pyocd"):
	if rtt_runner in ("jlink", "pyocd", "probe-rs"):

[MarkoSagadin]

Thank you for noticing, I have added probe-rs.