OTBR Agent timeout communicating with RCP (JN5189) during secondary commissioning in multi-admin mode

[Shangeev-AP]

Description:
We’re experiencing a failure in a Matter multi-admin commissioning setup using OTBR with JN5189 as the RCP. After a successful primary commissioning via the CHIPTool Android APK, the secondary commissioning using the CHIP Tool CLI fails. The OTBR Agent logs show timeouts while trying to communicate with the RCP, which causes the agent to stop functioning until a manual reset or RCP replug.

Setup:
Component Version / Description
Primary Commissioner : CHIPTool Android APK
Secondary Commissioner : CHIP Tool CLI (Linux)
Thread Device : Nanoleaf Bulb
RCP Platform : NXP JN5189 over UART (Baud Rates Tested : 115200 , 921600, 460800)
RCP Firmware Commit : dfbe12280af4a87cb9ae0c0c1d90d71de3372d82 (from ot-nxp)
OTBR Platform : Ubuntu 22.04 x86_64
OTBR Commit : b8bab1babcb040251b51a69ccb4008fb4b9e2d3f
CHIP Tool CLI Commit : 9af93fd3e9eab4315e1097b19112ebac01d1ad94

Steps to Reproduce:

1)Boot OTBR Agent with JN5189 RCP connected and initialized.

2. Perform primary commissioning using CHIPTool Android APK (success).

3. Open Enhanced commissioning window (multi-admin mode) via the APK.

4. Attempt secondary commissioning using CHIP Tool CLI.

5. Observe OTBR Agent behavior and logs.

Observation :

As per our observation, OTBR-Agent closes/stops running with RCP time out error under high network traffic.

I have attached the logs of OTBR-Agent and Secondary commisioner, CHIP Tool CLI

log_files.zip

[zhanglongxia]

The log shows that the Thread host failed to receive the response from the RCP. It seems that the issue comes from the RCP. I think you need to work with the chip vendor to check why the response is lost.

00:05:19.915 [W] P-RadioSpinel-: radio tx timeout
00:05:19.919 [C] P-RadioSpinel-: Failed to communicate with RCP - no response from RCP during initialization
00:05:19.919 [C] P-RadioSpinel-: This is not a bug and typically due a config error (wrong URL parameters) or bad RCP image:
00:05:19.919 [C] P-RadioSpinel-: - Make sure RCP is running the correct firmware
00:05:19.919 [C] P-RadioSpinel-: - Double check the config parameters passed as `RadioURL` input
00:05:19.919 [C] Platform------: HandleRcpTimeout() at radio_spinel.cpp:2035: RadioSpinelNoResponse

[Shangeev-AP]

What might be the most expected reasons for this behavior by the RCP?

[dsyx]

Under high network traffic, my OTBR (Raspberry Pi CM4 + Silabs MGM240 module) would sometimes trigger an RCP TX timeout and then reset:

Jun 25 06:38:52 CM4IO-MGM240 otbr-agent[706]: 15:07:10.607 [W] P-SpiIntface--: Garbage in header : 02 FB 01 FF FF
Jun 25 06:41:39 CM4IO-MGM240 otbr-agent[706]: 15:09:57.124 [W] P-RadioSpinel-: radio tx timeout
Jun 25 06:41:39 CM4IO-MGM240 otbr-agent[706]: 15:09:57.129 [W] P-RadioSpinel-: RCP failure detected
Jun 25 06:41:39 CM4IO-MGM240 otbr-agent[706]: 15:09:57.135 [W] P-RadioSpinel-: Trying to recover (1/100)
Jun 25 06:41:39 CM4IO-MGM240 otbr-agent[706]: 15:09:57.371 [C] P-RadioSpinel-: RCP => [C] Platform------: Reset info: 0x3 (EXT)
Jun 25 06:41:39 CM4IO-MGM240 otbr-agent[706]: 15:09:57.373 [C] P-RadioSpinel-: RCP => [C] Platform------: Extended Reset info: 0x301 (PIN)

[jwhui]

@Shangeev-AP @dsyx , out of curiosity, can you test with a different RCP platform, something like the nRF Dongle? This could help determine whether this is a RCP-specific issue.

[dsyx]

Hi @jwhui ,

I don't have an nRF Dongle at the moment, I only have Silabs' MGM240P, Espressif's ESP32-C6, and Nordic's nRF52840 DK.
I will try to use ESP32-C6 and nRF52840 DK as RCP later, but I have other work to do.

[dsyx]

Hi @jwhui , @Shangeev-AP ,

I have built a new OTBR (Raspberry Pi 4B + nRF52840 DK) following Set up the RCP Joiner and Native Install. I will report back here next Monday.

[jwhui]

For the nRF52840 RCP, you may also consider looking at the officially-supported method from the vendor.

[dsyx]

Nordic's approach and OpenThread's approach seem similar, are there any differences?

[jwhui]

The implementations are independent. Nordic's official solution is based on NCS, openthread/ot-nrf528xx is not.

[dsyx]

What is NCS? I am not very familiar with Nordic's development platform.

[jwhui]

NCS refers to Nordic Connect SDK, which is the officially supported SDK from Nordic.

[dsyx]

Hi @jwhui , @Shangeev-AP ,

After several days of high traffic testing, the official OTBR (Raspberry Pi 4B + nRF52840 DK) did not experience any RCP TX timeouts or resets. Does this look like a problem with other RCP vendor ports?

otbr-agent.zip

[jwhui]

I have seen this in other RCP implementations, but I don't have any data on how frequent or whether they have been fixed. As noted, any issues in the RCP that cause it to be unresponsive to the host should be fixed.

[dsyx]

This is a bit frustrating. It looks like only the Nordic platform has RCP implemented correctly at the moment.

[jwhui]

I wouldn't say Nordic is the only platform. For example, did you check Espressif?

[dsyx]

Since I have some other work to do right now, I don't have time to study Espressif-based OTBR for the time being. In the future, I will try to build OTBR using Espressif for testing.