Errors and Crashing When Using odrive-botwheeler with Plotjuggler

[Mohamed-Abduljawad]

I'm using the botwheeler ros2 control demo with plotjuggler to visualize the /gains/left_wheel_joint/pid_state topic, (in fact any related topic). Plotjuggler is subscribing to the topic and plotting it, however it eventually crashes, and the ros2 control node starts spamming
[ros2_control_node-1] Failed to send CAN frame And the communication stops.

I'm running the ros nodes in a docker running jazzy. While plotjuggler is running on main machine on Humble. This shouldn't cause a problem because previously I used to run my own ros2 control nodes and other topics and display them just fine in plotjuggler (Not using CAN bus though).

Any idea why this is happening? Please let me know if you need more info or logs data.

[samuelsadok]

This could mean that the CAN interface went down (e.g. USB-CAN adapter unplugged, "bus off" condition) or that the send buffer became full because the outgoing CAN messages weren't acknowledged by any CAN node.

You can get the lower level error code if you look for this line:

std::cerr << "Failed to send CAN frame" << std::endl;

in socket_can.cpp and replace it with this:

int error_code = errno;
std::cerr << "Failed to send CAN frame: " << std::strerror(error_code) << " (" << error_code << ")" << std::endl;

[Mohamed-Abduljawad]

I added that snippet and got this error:
[ros2_control_node-1] Failed to send CAN frame: No buffer space available (105)

Also after multiple tries, it is not always showing as I stated previously, it seems somewhat random. But what I'm sure about it is that plotjuggler uses all the memory then crashes, and the ros node throws this error simultaneously.

For reference this is the terminal error where I'm running snap version of plotjuggler 3.13.2:

  ''Bad alloc' exception deserializing message of type rmw_dds_common::msg::dds_::ParticipantEntitiesInfo_., at ./src/type_support_common.cpp:123'

with this new error message:

  'Fast CDR exception deserializing message of type rmw_dds_common::msg::dds_::ParticipantEntitiesInfo_., at ./src/type_support_common.cpp:118'

rcutils_reset_error() should be called after error handling to avoid this.```

Even if not crashing, plotjuggler stops receiving and plotting topics when I start publishing/subscribing to similar botwheels topics. 

[samuelsadok]

"No buffer space available" is typically indicative of:

send buffer became full because the outgoing CAN messages weren't acknowledged by any CAN node

However you say that it happens at the same time as an out-of-memory crash of plotjuggler. It's slightly unexpected to me that this affects the CAN socket because the SocketCAN send queue has a user-defined capacity so I would have thought the kernel would pre-allocate memory for it. But fundamentally it's totally possible that it also uses just-in-time allocation despite the fixed queue size. So that would be an explanation.

In this case I recommend to review memory usage in your system. It sounds like a memory leak somewhere. Identifying the culprit should be easy with standard monitoring tools like htop.