NCP host doesn't free used TID after factoryreset CLI command

[bertoldi-silabs]

Describe the bug Issuing the factoryreset CLI command repeatedly on a border router with an NCP secondary will eventually result in the CLI session losing responsiveness until the service is restarted.

To Reproduce Open up a CLI session to a border router with an NCP secondary using ot-ctl. Issue factoryreset 15 times, and then attempt to issue additional commands. The CLI spinel frame will fail to encode with an OT_ERROR_BUSY error.

1. Git commit id - 05c8ae8
2. IEEE 802.15.4 hardware platform - raspberry pi 4 + efr32 (SiLabs)
3. Build steps - ./script/cmake-build -DOT_THREAD_VERSION=1.4
   -DOT_MULTIPAN_RCP=ON
   -DCPCD_SOURCE_DIR=
   -DOT_POSIX_RCP_HDLC_BUS=ON
   -DOT_POSIX_RCP_SPI_BUS=ON
   -DOT_POSIX_RCP_VENDOR_BUS=ON
   -DOT_POSIX_CONFIG_RCP_VENDOR_DEPS_PACKAGE=
   -DOT_POSIX_CONFIG_RCP_VENDOR_INTERFACE= <cpc_interface.cpp>
   -DOT_PLATFORM_CONFIG=openthread-core-silabs-posix-config.h
   -DOTBR_DHCP6_PD=ON
4. Network topology - one node: respberry pi 4 + efr32 (SiLabs)

Expected behavior Repeatedly issuing the factoryreset CLI command should not cause CLI unresponsiveness. Alternately, the NCP resetting should free any used TIDs to reduce TID starvation in future spinel exchanges.

Additional context There are other scenarios in which a high amount of spinel traffic can cause TID starvation that leads to the CLI session being unresponsive. NetIF is an area of concern; however, details for this symptom are still being evaluated.

[bertoldi-silabs]

Hello @Irving-cl. I was discussing this issue with @hemanth-silabs, and he mentioned that I should create an issue for this topic to be further discussed.

I am observing more situations than just the factoryreset which lead to TID starvation, but further understanding is needed before I will share any claims about them. I will provide updates here when I do have more to share. Thanks for your interest.

[suveshpratapa]

cc @Irving-cl and @abtink ... we need some input here on how exactly the design for "factoryreset" on a host/NCP architecture should work. In the current implementation, the CLI command is processed on the NCP and the host is still active while it waits for last status, leading to some issues enumerated above.

[Irving-cl]

@suveshpratapa We require further design work on this matter. I need some additional time to consider the details. Will try to do this soon and get back to you.

[Irving-cl]

Hi @bertoldi-silabs @suveshpratapa

Thanks for identifying the problem. Upon further reflection, my first step is to implement the currently missing logic for managing unexpected NCP resets. While this functionality exists in RcpSpinel, it has not yet been integrated into NcpSpinel. I suspect that NCP crashes are occurring where the NCP reboots without the host's knowledge, potentially causing tid leaks.

To address RCP crashes, there are generally two strategies:

• The otbr-agent can be configured to abort alongside the crash, allowing the product's service system to trigger a restart.
• RCP restoration can be attempted, where the otbr-agent remains active and tries to recover the RCP. This method is significantly more intricate and is seldom utilized in production environments.

I intend to implement the first strategy for NCP immediately. Furthermore, we must investigate the root cause of these NCP crashes. I am preparing a PR to include NCP crash handling; please test the system again once that is submitted to see if any crashes happen.

Regarding the use of 'factoryreset', I assume you are using it to see if it clears the issues you've encountered. However, a 'factoryreset' triggers an NCP restart, which leads back to the aforementioned situation. While adding formal support for a factoryreset CLI command is a reasonable goal that I will look into, please attempt to replicate the problem without using factoryreset for the time being.

[suveshpratapa]

Hi @Irving-cl I've taken a look at the PR. Thank you!

For resets, we have two cases:

• A. Unexpected NCP resets or crashes that host cannot anticipate: In this case, your PR to terminate the host seems appropriate.
• B. Intentional factoryreset (user-initiated) on host: Since this is a deliberate action, we think the host should be explicitly aware this is happening. In our case, we had a test procedure that was doing this action. For now, we are fine going with an approach to manually have our test procedure terminate the host along with issuing a factoryreset.

One approach we discussed for case B is that the host could ideally intercept the command/API on the host before sending to NCP, so it can perform some orderly cleanup, such as flushing TID tables. We could also automatically terminate the host to keep it consistent with the case A above.

I think we're also trying to understand the TID starvation issue a bit further, if it's simply due to high traffic volume or some other bug. cc @bertoldi-silabs please correct me if I'm wrong.

[Irving-cl]

@suveshpratapa

One approach we discussed for case B is that the host could ideally intercept the command/API on the host before sending to NCP, so it can perform some orderly cleanup, such as flushing TID tables. We could also automatically terminate the host to keep it consistent with the case A above.

Yes, that's also something I'm thinking. Will follow it up.