Possible memory leak with Gemini 435Le on v2.4.7

[dimatura]

Hello, we recently upgraded the OrbbecSDK_ROS2 wrapper to the latest version (v2.4.7) of the ROS2 wrapper, and immediately started seeing a runaway memory usage increase on our system when using two Gemini 435Le cameras.
We use an unsynced multi camera launchfile setup based on the example on this repo. As soon as the launchfile starts, the nodes operate normally, but RAM usage quickly climbs and within minutes fills up the whole 16GB ram available, unless we kill it. With the previous release of OrbbecSDK_ROS2, RAM usage is under 2%.

Some more details:

• The host system is a Jetson Orin NX, 16GB, Jetpack 6.2
• Ubuntu is 22.04 jammy and ROS2 is humble.
• The problem occurs whether we use the v3.02 of the Gemini 435Lefirmware or the latest, v3.06.
• We use the default stock configuration, no parameter changes.
• We deploy OrbbecSDK_ROS2 by using a .deb built file using .make_deb.sh script provided in this repo. This worked fine with the v2.4.5 release. (Update: I also tried the .deb file from the github release page, the behavior was the same).
• Logs for the nodes don't show any errors. (Update: using a debug log level does show errors, I've added logs below).

One possible clue is that investigating the problem, I tried adding "enumerate_net_device": "false" to the launch parameters (for no specific reason, just trying things), and surprisingly, that seemed to help. Launching and stopping the nodes with this change seemed to bring memory usage back to normal, except one time where it started leaking again, but not 100% sure about that one time (I was changing a few things when testing). We're currently testing more with this change and I can update, but regardless, it definitely seems like there's some buggy behavior here.

Update: it seems like enumerate_net_device: "false" only delays the memory leak, after running the launch file for a few more minutes RAM starts blowing up again.

Second update: if I start with enumerate_net_device: "false", memory usage seems to be low and relatively stable, until I disconnect the host's device ethernet cable to the switch to which the two gemini 435le cameras are also connected - then memory usage starts rapidly going up again, even after reconnecting the cable.

[dimatura]

The full debug logs did not get written to disk, but monitoring the console output, I can see that at the time when memory starts to increase the following exception gets reported continuously:

[component_container-1] [10/07 03:14:21.451638][warning][3451][ObException.hpp:48] VendorTCPClient write data failed! socket=44, err_code=32
[component_container-1] [10/07 03:14:21.452934][warning][3451][ObException.hpp:48] Request failed, statusCode: 2, msg: Send control transfer failed!
[component_container-1] [10/07 03:14:21.453174][warning][3451][SensorBase.cpp:363] Execute failure! A libobsensor_exception has occurred!
[component_container-1]  - where: SensorBase.cpp(363):outputFrame
[component_container-1]  - msg: Request failed, statusCode: 2, msg: Send control transfer failed!
[component_container-1]  - type: N11libobsensor12io_exceptionE
[component_container-2] [10/07 03:14:21.529719][debug][3225][PropertyServer.cpp:258] Property 1037 get structure data successfully, size 16
[component_container-2] [10/07 03:14:21.529833][debug][3225][GlobalTimestampFitter.cpp:136] Get device time rtt is too large! rtt=1508556
[component_container-1] [10/07 03:14:21.546444][warning][3451][ObException.hpp:48] VendorTCPClient write data failed! socket=44, err_code=32
[component_container-1] [10/07 03:14:21.546551][warning][3451][ObException.hpp:48] Request failed, statusCode: 2, msg: Send control transfer failed!
[component_container-1] [10/07 03:14:21.546654][warning][3451][SensorBase.cpp:363] Execute failure! A libobsensor_exception has occurred!
[component_container-1]  - where: SensorBase.cpp(363):outputFrame
[component_container-1]  - msg: Request failed, statusCode: 2, msg: Send control transfer failed!
[component_container-1]  - type: N11libobsensor12io_exceptionE
[component_container-1] [10/07 03:14:21.655912][warning][3451][ObException.hpp:48] VendorTCPClient write data failed! socket=44, err_code=32
[component_container-1] [10/07 03:14:21.656022][warning][3451][ObException.hpp:48] Request failed, statusCode: 2, msg: Send control transfer failed!
[component_container-1] [10/07 03:14:21.656132][warning][3451][SensorBase.cpp:363] Execute failure! A libobsensor_exception has occurred!
[component_container-1]  - where: SensorBase.cpp(363):outputFrame
[component_container-1]  - msg: Request failed, statusCode: 2, msg: Send control transfer failed!
[component_container-1]  - type: N11libobsensor12io_exceptionE

[dimatura]

Here is a full log with debug messages. I actually believe the memory starts leaking slowly from the start, but at around midway through the logs (line 33000 or so) exceptions start happening continuously, and memory usage blows up until the node gets killed by the kernel due to OOM.
gemini435le_memory_leak.txt.gz

[obshenlanzhou]

@dimatura Hi, thanks for your feedback!
We are currently trying to resolve this issue.

[ob-yalian]

@dimatura Hi. We have tested the same setup on multiple devices and environments but were not able to reproduce the memory leak issue you described.

To help narrow down the cause, we recommend trying the following:

• Run the setup on a different system or environment to see whether the issue persists.
• Try using the latest v2-main branch that we have just uploaded — it includes several updates and fixes that might address this behavior.

Please let us know if the issue still occurs under the new environment or with the latest branch.
Thank you for your continued feedback and support.

[dimatura]

Thanks, I will try again and report back.