SDO writes reply are intermittently slow

[chen27040604-cell]

In a single SDO client to multiple SDO server setup, it is noted that the reply to SDO writes can take an extended amount of time for nodes. The issue does not present every time, but can be replicated consistently, and is persistent to that node(s) once replicated until reset.

This issue usually affects all replies to SDO writes for that specific node. A reply to an SDO write in nominal operations will take <1ms, but with this issue, the reply takes closer to ~1 second.

Is there a reason a reply to an SDO write can take so long, which is close to the default SDO write abort timeout? This is considering that there are barely any other messages on the bus except for heartbeats.

Here is a scenario of the issue:
We have 3 CANopen nodes, 1 acting as the SDO client and the other 2 acting as individual SDO servers (SDO server 1 and SDO server 2).
A function is called to execute several SDO writes to SDO server 1 and then subsequently several SDO writes to SDO server 2.
In this scenario let's say SDO server 2 presents with the issue.
The SDO client is able to execute the SDO writes and receive a reply from SDO server 1 within a millisecond for each SDO write (as witnessed by a CAN sniffer).
The SDO client is able to execute the SDO writes to SDO server 2, but the reply now takes closer to 1 second for each SDO write.

Other observations to note:
The issue presents intermittently, so a scenario can have all SDO server nodes performing nominally, but the issue can be forced by repeatedly resetting the node through the NUCLEO reset button. The issue can also be remedied by resetting the affected nodes until the SDO writes are "fast" again. Although over time the SDO servers will consistently become "slow" with sending replies to SDO writes. They will not go back to nominal speeds until the node is reset.

Here are configurations/setup details as well as some notes about certain setups:
SDO client is on a NUCLEO-F439ZI board
SDO servers are on NUCLEO-G0B1RE boards
canopen_app_interrupt on a 1ms timer (extending to 2ms does help the issue not occur as often but does not solve it)
canopen_app_process called by FreeRTOS task with a task delay of 1ms
CAN bus speed set to 1Mbps
When the NUCLEO boards are powered off of STLINK (micro USB) and the CAN transceivers are powered off of the NUCLEO 5V output, the issue occurs less often on powerup. But if the CAN transceivers are not powered off the NUCLEO 5V output and instead on a separate external power supply, the issue occurs every time on powerup for all nodes with that configuration.

Side question:
Is there a reason the write_SDO and read_SDO functions have a 10ms delay in the do while loop? Is this purely for stability? Can it perform reliably if say this delay was lowered down to 1ms or even 100us?

[CANopenNode]

CAN dump with timestamp could be very helpful here. It should show the source of delays.

There could be a problem with integration of CANopenNode into mainline, RTOS, etc. Probably not driver(STM32) issues. However, SDO should have super fast response, see candump example in https://github.com/CANopenNode/CANopenDemo/blob/master/tutorial/SDO.md.

Those loops may have delays and there is no need to use very short delays. It only consumes power. Make sure, you use timeDifference_us and timerNext_us variables correctly inside mainline or RTOS.

[chen27040604-cell]

Thank you for the reply! I am using Savvy CAN with a CSS electronics CL1000 tool to view the raw CAN messages with timestamps. The results are what led me to believe that the significant delay comes from the SDO server reply for the affected node(s).

Here is a snippet of a capture with the first node behaving nominally and the next node exhibiting the delayed response behavior. The SDO client message to the first node (0x603 ID) is met with very fast responses (~1ms) by the first SDO server node (0x583 ID), as desired. The problem is starting at row 29, each SDO client message to the second node (0x604 ID) is met with varying delayed responses (100 ms to 750 ms) by the second SDO server node (0x584 ID).

Verifying the hardware setup (checked terminating resistors, CAN H and L line, etc.), experimenting with the CAN/FDCAN configuration (to note the G0B1RE boards (SDO servers) support FDCAN but are configured to classic mode frame format and the F439ZI boards (SDO client) support only classic CAN), and debugging the code to note any differences in the step through of the SDO server process between a nominal and delayed node resulted in no resolution to the issue.

The issue is also exacerbated with the addition of more SDO server nodes, where with a 1 to 1 SDO client to server connection nearly never has the delay issue (have not witnessed), but 1 to 2 and 1 to 3 increasingly presents with the delay issue more often. Also to reiterate, even if all SDO server nodes are behaving nominally at power up, at least 1 node over time will eventually have the delay issue become persistent until manual reset.

One observation to note is that the canopen_app_interrupt callback was taking nearly 900us to complete from OD lock to OD unlock from a logic analyzer, which is what prompted extending the extension of the timer from 1ms to 2ms. This did seemingly help with the issue not presenting as often (again small sample sizes) but did not resolve it with even extending the timer to 5ms.

Also sync jump width was extended to 4 time quanta to see if that helps, but it did not.

Any insight or things to try will help and will be greatly appreciated, if you need any more info, please ask! Thanks again!