[Humble] ROS2 service call for DeviceReset

[Likhith-Sugganahalli]

Hi,

Thanks a lot for your work on this project.

I am using a blackfly_s and have been encountering an issue with the FPS suddenly dropping to ~2 on a start, out of the blue, while it is generally ~16. Whenever this issue happens, it also reflects in the SpinView application and the only solution i have found is to unplug the camera and plug it back in.

Hence, i wanted to check if i can reset the camera from code itself, i see that the spinnaker API lists DeviceReset, but i am unable to find it being used in this project, is there any other way i can accomplish this?

Thanks

[berndpfrommer]

I added a service call to the driver to call the DeviceReset, but it's obviously not that easy:

[camera_driver_node-1] [INFO] [1738243628.610429859] [20435008]: rate [Hz] in  40.00 out   0.00 drop   0%
[camera_driver_node-1] WARNING: acquisition timeout, restarting!
[camera_driver_node-1] terminate called after throwing an instance of 'Spinnaker::Exception'
[camera_driver_node-1]   what():  Spinnaker: Camera is disconnected [-1014]
[ERROR] [camera_driver_node-1]: process has died [pid 59396, exit code -6, cmd 

Rebooting the camerat while keeping the driver running is tedious to develop, and in the end it's a band-aid to work around an issue without understanding the root cause of it.
Can you say more about your configuration (what camera model)? There is a reason why I added the issue template...

[Likhith-Sugganahalli]

@berndpfrommer

Heyya, sorry for not using the template, adding the information here.

Is this behavior also observed when using SpinVIew?
The bug where out of the blue the driver starts with low FPS is also observed when i run the SpinView (after having observed the issue with the ros2 driver). When i execute the DeviceReset from the SpinView, it starts running at the correct FPS, also works when i unplug the usb cable and put it back in.

Describe the feature
A service call that executes the DeviceReset as seen in the SpinView

Useful logging output
Do not have any useful logs for this

System details

• operating system/version: Ubuntu 22.04
• ROS version (please prepend to issue title in square brackets, e.g. [noetic]): humble running the humble-devel branch
• complete FLIR camera model (e.g. BFS-U3-04S2C-C): BFS-U3-200S6M

[berndpfrommer]

Can you check out PR #227 ? It does reboot the camera. For me however that changes the frame rate because apparently some settings are not defaulted.

But back to your problem. I see you have a very high resolution camera. Check the quality of your USB3 cable, and for errors via journalctl -f

Is the camera externally triggered or free running? Is auto exposure switched on? If the exposure time is longer than 1 / fps the frame rate will drop, so beware of that. Disable auto exp and see if that fixes the problem.

[Likhith-Sugganahalli]

@berndpfrommer I will try out the PR, thanks for that.

I do have a good quality usb3 cable, i was able to verify using syslogs that the usb connection was for super high speed or something of that sort.

The camera is free running, exposure time is set much lower than 1/fps.

[berndpfrommer]

You should see errors somewhere if the USB cable is bad. I only use FLIR cables b/c the cheap cables did not work for me.

I don't ever recall a case where a FLIR camera would spontaneously change frame rate, except for exposure time related issue. Could the camera be overheating? Do you see the same drop in frame rate when you run on SpinView for a while?

[Likhith-Sugganahalli]

@berndpfrommer
Yeah, this issue has stumped flir support as well, there is no error of any sort, in the ros2 driver log, it says that it's starting with fps of 19, but when the acquisition logs start printing, it's 2 fps instead.

I am using flir cable as well,no overheating as well

I do see the same issue with SpinView, but then, the application loads the ros2 driver and when there's this fps drop, I check again with SpinView, so I can't say if this drop wouldn't happen if I only ran SpinView.

Also, this issue doesn't seem to be related to how long I run the system, I haven't been able to spot any pattern to when the issue pops up.

Do you know where the SpinView saves exhaustive logs on Ubuntu? I found some logs in .config folder, but they seem to be useless, maybe if I catch this issue in SpinView and check the logs, it might shed some light on it.

Thanks a lot for your help, really appreciate it!

[berndpfrommer]

Hmm. You have a second camera to play with? Usually FLIR cameras are very reliable, but this could be a bad apple.

Does the drop only happen on ROS2 driver startup, never while the driver has been running a while? Then it could be a race condition when the parameter setting is passed to the camera. Not sure exactly how that would be able to screw up the camera to the point that you need to power cycle, but that would be my hunch then. I remember when developing the ROS2 driver's user setting load feature that I screwed cameras up badly, to the point they needed power cycle and firmware reflash. Are you loading back user settings by any chance?

I didn't even know about the SpinView logs. I already moved on to Ubuntu 24.04 (noble), so I haven't been running SpinView lately.

[Likhith-Sugganahalli]

@berndpfrommer

Heyya,

I do not have a second camera to play with, i will see if i can try something out there.

the drop only happens at startup, never while the driver has been running a while. I am not loading user settings, i am explicitly setting each parameter, i will check if maybe the order in which I am setting them makes it more likely that this happens once in a while.

I will test out the DeviceReset service call and update on this thread shortly.

[berndpfrommer]

If this fps drop always happens at startup, then almost certainly it has to do with the ROS2 driver somehow setting one parameter wrong, or the SDK having a bug that screws up the parameter in the camera memory.
I don't think resetting the device from the ROS2 driver is a good idea. What I implemented so far does not re-apply the parameters so the camera will come up in a different state than before the reset, and you'll have to set the parameters again manually. I don't want to add more complexity to the driver to work around this issue.
Do your camera parameters stay the same? Then try the "user settings" feature of the camera: in SpinView, set up the parameters to your liking, then save them as a user setting. With the ROS2 driver, don't set any parameters but load the user settings (there is a parameter for that, see the driver_node.launch.py file.
OOPS: Just saw that the default launch file loads user set 0. Do you have that enabled? That could cause a race condition and possibly mess up the camera. If you have that enabled, please try first removing the loading of the user set.

[Likhith-Sugganahalli]

@berndpfrommer

hi,

I was able to verify that the service call actually does help with the issue.

at a fresh start of the PC, when i ran the launch file, i started getting 2 fps, restarting the PC did not solve the issue, so i checked out support_device_reset branch, built the driver, re-ran the launch file, it still ran at 2 fps, when i called the reset service call, the issue was resolved.

[INFO] [1738747661.688731815] [22623936]: reading ros parameters for camera with serial: 22623936
[INFO] [1738747661.688764345] [22623936]: adjusting time stamps!
[INFO] [1738747661.688800079] [22623936]: parameter definitions file: /home/user/arm_ws/install/spinnaker_camera_driver/share/spinnaker_camera_driver/config/blackfly_s.yaml
[INFO] [1738747661.689933934] [flir_camera]: using default calibration URL
[INFO] [1738747661.689961738] [flir_camera]: camera calibration URL: file:///home/user/.ros/camera_info/flir_camera.yaml
[ERROR] [1738747661.689981423] [camera_calibration_parsers]: Unable to open camera calibration file [/home/user/.ros/camera_info/flir_camera.yaml]
[WARN] [1738747661.689984789] [flir_camera]: Camera calibration file /home/user/.ros/camera_info/flir_camera.yaml not found
[INFO] [1738747662.673720789] [22623936]: using spinnaker lib version: 4.0.0.116
[INFO] [1738747662.673876417] [22623936]: found camera with serial number: 22623936
[INFO] [1738747662.904122655] [Spinnaker Wrapper]: Initialized camera [serial: 22623936] from: [USB Interface 0]
[INFO] [1738747662.904223688] [22623936]: setting ImageFormatControl/PixelFormat to: Mono8
[INFO] [1738747662.910850547] [22623936]: setting ImageFormatControl/BinningHorizontal to: 1
[INFO] [1738747662.917617013] [22623936]: setting ImageFormatControl/BinningVertical to: 1
[INFO] [1738747662.924534673] [22623936]: setting ImageFormatControl/Width to: 5472
[INFO] [1738747662.928035446] [22623936]: setting ImageFormatControl/Height to: 3648
[INFO] [1738747662.931352860] [22623936]: setting ImageFormatControl/OffsetX to: 0
[INFO] [1738747662.934848651] [22623936]: setting ImageFormatControl/OffsetY to: 0
[INFO] [1738747662.938271601] [22623936]: setting AnalogControl/GainAuto to: Continuous
[INFO] [1738747662.941019706] [22623936]: setting DeviceControl/DeviceLinkThroughputLimit to: 396465700
[INFO] [1738747662.943535588] [22623936]: setting AcquisitionControl/ExposureAuto to: Off
[INFO] [1738747662.946156247] [22623936]: setting AcquisitionControl/ExposureTime to: 100
[INFO] [1738747662.947353241] [22623936]: setting AcquisitionControl/AcquisitionFrameRateEnable to: 1
[INFO] [1738747662.949282515] [22623936]: setting AcquisitionControl/AcquisitionFrameRate to: 40
[WARN] [1738747662.950117722] [22623936]: param frame_rate Spinnaker: GenICam::OutOfRangeException= Value 40.000000 must be smaller than or equal 19.861002. : OutOfRangeException thrown in node 'AcquisitionFrameRate' while calling 'AcquisitionFrameRate.SetValue()' (file 'FloatT.h', line 85) [-2002]
[INFO] [1738747662.950134693] [22623936]: setting AcquisitionControl/TriggerMode to: Off
[INFO] [1738747662.952626126] [22623936]: setting ChunkDataControl/ChunkModeActive to: 1
[INFO] [1738747662.954802275] [22623936]: setting ChunkDataControl/ChunkSelector to: FrameID
[INFO] [1738747662.955735395] [22623936]: setting ChunkDataControl/ChunkEnable to: 1
[INFO] [1738747662.957723345] [22623936]: setting ChunkDataControl/ChunkSelector to: ExposureTime
[INFO] [1738747662.958103523] [22623936]: setting ChunkDataControl/ChunkEnable to: 1
[INFO] [1738747662.959876931] [22623936]: setting ChunkDataControl/ChunkSelector to: Gain
[INFO] [1738747662.960260894] [22623936]: setting ChunkDataControl/ChunkEnable to: 1
[INFO] [1738747662.962027541] [22623936]: setting ChunkDataControl/ChunkSelector to: Timestamp
[INFO] [1738747662.962288184] [22623936]: setting ChunkDataControl/ChunkEnable to: 1
[INFO] [1738747662.984145685] [22623936]: camera has pixel format: Mono8
[INFO] [1738747662.984168054] [flir_camera]: camera_driver_node started up!
[INFO] [1738747666.689017326] [22623936]: rate [Hz] in 2.15 out 0.00 drop 0%
[INFO] [1738747671.688994263] [22623936]: rate [Hz] in 2.15 out 0.00 drop 0%
[INFO] [1738747676.688801484] [22623936]: rate [Hz] in 2.15 out 0.00 drop 0%
[INFO] [1738747681.688983350] [22623936]: rate [Hz] in 2.15 out 0.00 drop 0%
[INFO] [1738747686.688933402] [22623936]: rate [Hz] in 2.15 out 0.00 drop 0%
[INFO] [1738747691.689014103] [22623936]: rate [Hz] in 2.15 out 0.00 drop 0%
[INFO] [1738747694.010898583] [22623936]: restarting camera in 10 seconds
[INFO] [1738747696.689004148] [22623936]: rate [Hz] in 2.15 out 0.00 drop 0%
[INFO] [1738747701.689013716] [22623936]: rate [Hz] in 2.15 out 0.00 drop 0%
[INFO] [1738747704.033996741] [22623936]: camera has pixel format: Mono8
[INFO] [1738747706.688997852] [22623936]: rate [Hz] in 2.56 out 0.00 drop 0%
[INFO] [1738747711.689056857] [22623936]: rate [Hz] in 4.91 out 0.00 drop 0%
[INFO] [1738747716.688946892] [22623936]: rate [Hz] in 8.16 out 0.00 drop 0%
[INFO] [1738747721.689005989] [22623936]: rate [Hz] in 11.46 out 0.00 drop 0%
[INFO] [1738747726.689010470] [22623936]: rate [Hz] in 13.90 out 0.00 drop 0%
[INFO] [1738747731.688955430] [22623936]: rate [Hz] in 15.32 out 0.00 drop 0%
[INFO] [1738747736.689010959] [22623936]: rate [Hz] in 16.02 out 0.00 drop 0%
[INFO] [1738747741.689017884] [22623936]: rate [Hz] in 16.35 out 0.00 drop 0%
[INFO] [1738747746.688959111] [22623936]: rate [Hz] in 16.49 out 0.00 drop 0%
[INFO] [1738747751.689013561] [22623936]: rate [Hz] in 16.56 out 0.00 drop 0%
[INFO] [1738747756.688985401] [22623936]: rate [Hz] in 16.59 out 0.00 drop 0%
[INFO] [1738747761.689020007] [22623936]: rate [Hz] in 16.60 out 0.00 drop 0%
[INFO] [1738747766.689078179] [22623936]: rate [Hz] in 16.60 out 0.00 drop 0%
[INFO] [1738747770.002727911] [rclcpp]: signal_handler(signum=2)

i am not using the user setting, have that commented out.Below is the parameters i have defined in a yaml which i load in the launch file and pass as parameters to the driver node.

debug: False
compute_brightness: False
adjust_timestamp: True
dump_node_map: False
gain_auto: "Continuous"
pixel_format: "Mono8"
exposure_auto: "Off"
exposure_mode: "Timed"
exposure_time: 100.0
image_height: 3648
image_width: 5472
offset_x: 0
offset_y: 0
sensor_shutter_mode: "Rolling"
binning_selector: "All"
binning_x: 1
binning_x_mode: "Sum"
binning_y: 1
binning_y_mode: "Sum"
frame_rate_auto: "Off"
frame_rate: 40.0
frame_rate_enable: True
trigger_mode: "Off"
connect_while_subscribed: False
enable_external_control: False
chunk_mode_active: True
chunk_selector_frame_id: "FrameID"
chunk_enable_frame_id: True
chunk_selector_exposure_time: "ExposureTime"
chunk_enable_exposure_time: True
chunk_selector_gain: "Gain"
chunk_enable_gain: True
chunk_selector_timestamp: "Timestamp"
chunk_enable_timestamp: True
device_link_throughput_limit: 396465700

i would like to try to get the reset time down from 10 seconds, any suggestions on that?

[berndpfrommer]

Why do you set the FPS to 40 when it can only support 19.86 (see error message in log)?