IPv6 Queue Buffer filling up and blocking node operation while using CoAP

[FGK42]

Hi, I would really appreciate your help on this subject.

I have a node transmitting a CoAP non-confirmable POST message every 10 seconds, with a payload of 84 bytes.

It generally runs properly, using a maximum of 13-18 buffers. I have a total of 176 buffers available.

I print the state of the buffers every second, and in general the IPv6 Queue buffers/messages are always 0.

After a random time, generally a few hours, the IPv6 queue buffers jumps to 2 (from 1 message consisting of 132 bytes), and then starts increasing rapidly by increments of 1 message. Filling up the buffer rapidly and then blocking all operations of the node.

I checked, and the 132 bytes corresponds to the usual IPv6 UDP packet size.

**The issue starts with the transmission attempt of a CoAP message.
During normal operation, this message is followed by MAC 'request to start operation...' etc. But now nothing happens.
After that, I can see the IPv6 queue buffer messages has incremented by 1.

After that, every message attempt, including advertisements, increment the IPv6 message queue.

During this period of incrementing IPv6 message buffers, I still succesfully receive time sync IPv6 UDP messages, indicating that the reception works fine.

To me it seems that there is some failure in transmission After the failure, all the transmission attemps (advertising and sending coap) do not process and fill the IPv6 message queue. I check for errors during CoAP message creation, and the code receives an OT_ERROR_NONE after calling otCoapSendRequest(), indicating that this should be fine.

Does anyone have an indication or clue to what could be the issue, or where I could focus on debugging?

Kind regards,

[abtink]

@FGK42 , we have observed similar situations in the past, and they are almost always due to issues or bugs at the radio platform layer.

Essentially, the radio driver fails to signal otPlatRadioTxDone after a successful otPlatRadioTransmit. This causes the OpenThread MAC layer (and/or the RCP) to get stuck, which in turn causes IPv6 messages to pile up in the transmit queues (at the MeshForwarder).

I would suggest either investigating the radio platform layer or reaching out to the radio vendor to check on this.

[FGK42]

Thank you for the reply @abtink. I've contacted the vendor and will look at the radio platform layer. I'll get back to you when I have more information.

[FGK42]

Hi @abtink,

I found the issue, which was not an issue with the radio platform, but more an oversight on my part which causes the weird behavior:

Cause:
I sent COAP transmissions by calling the function in the interrupt service routine called by the timer

Fix:
I moved the coap transmission function to the main loop, which now checks if the transmission flag is set
The interrupt service routine now only takes a measurement and sets the transmission flag

I guess that the ISR, even when at lowest priority, still interrupted OT / radio processes. Adding another message transmission during this interrupt caused unexpected behavior and caused these issues.

I still find it curious that all debug related info from OT / the platform indicated no issues concerning the CoAP transmission. The return message from OT on requesting the transmission of the new COAP message was still ' OT_ERROR_NONE'., but I'm glad I found the issue.

Anyway, thanks for your suggestion.