Missing attribute `self._tasks` when running executor `AutoScalingMultiThreadedExecutor.shutdown()` in ROS 2 Jazy

[zmk5]

Bug description

Platform:

• OS: Ubuntu Noble
• ROS: Jazzy
• proto2ros version: Not Sure

How to reproduce

I currently followed this Dockerfile to build spot_ros2 in jazzy. I went through the installation and it was successful. However, when attempting to run the spot_driver via the launch file (ros2 launch spot_driver spot_driver.launch.py) I noticed two nodes failing,spot_alerts and spot_ros2, with the same failure:

[spot_alerts-6] Traceback (most recent call last):
[spot_alerts-6]   File "/spot_ws/install/spot_driver/lib/spot_driver/spot_alerts", line 56, in <module>
[spot_alerts-6]     main()
[spot_alerts-6]   File "/spot_ws/install/synchros2/lib/python3.12/site-packages/synchros2/process.py", line 190, in __call__
[spot_alerts-6]     with scope.top(argv, global_=True, interruptible=self._interruptible, **scope_kwargs) as self._scope:
[spot_alerts-6]   File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__
[spot_alerts-6]     self.gen.throw(value)
[spot_alerts-6]   File "/spot_ws/install/synchros2/lib/python3.12/site-packages/synchros2/scope.py", line 556, in top
[spot_alerts-6]     stack.close()
[spot_alerts-6]   File "/usr/lib/python3.12/contextlib.py", line 618, in close
[spot_alerts-6]     self.__exit__(None, None, None)
[spot_alerts-6]   File "/usr/lib/python3.12/contextlib.py", line 610, in __exit__
[spot_alerts-6]     raise exc_details[1]
[spot_alerts-6]   File "/usr/lib/python3.12/contextlib.py", line 595, in __exit__
[spot_alerts-6]     if cb(*exc_details):
[spot_alerts-6]        ^^^^^^^^^^^^^^^^
[spot_alerts-6]   File "/spot_ws/install/synchros2/lib/python3.12/site-packages/synchros2/scope.py", line 165, in __exit__
[spot_alerts-6]     self._stack.close()
[spot_alerts-6]   File "/usr/lib/python3.12/contextlib.py", line 618, in close
[spot_alerts-6]     self.__exit__(None, None, None)
[spot_alerts-6]   File "/usr/lib/python3.12/contextlib.py", line 610, in __exit__                                                                                                                                     [spot_alerts-6]     raise exc_details[1]
[spot_alerts-6]   File "/usr/lib/python3.12/contextlib.py", line 595, in __exit__
[spot_alerts-6]     if cb(*exc_details):
[spot_alerts-6]        ^^^^^^^^^^^^^^^^
[spot_alerts-6]   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
[spot_alerts-6]     next(self.gen)
[spot_alerts-6]   File "/spot_ws/install/synchros2/lib/python3.12/site-packages/synchros2/executors.py", line 837, in foreground
[spot_alerts-6]     if not executor.shutdown(timeout_sec=5.0):
[spot_alerts-6]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[spot_alerts-6]   File "/spot_ws/install/synchros2/lib/python3.12/site-packages/synchros2/executors.py", line 768, in shutdown
[spot_alerts-6]     AutoScalingMultiThreadedExecutor.Task(task, entity, node) for task, entity, node in self._tasks
[spot_alerts-6]                                                                                         ^^^^^^^^^^^
[spot_alerts-6] AttributeError: 'AutoScalingMultiThreadedExecutor' object has no attribute '_tasks'
[ERROR] [spot_alerts-6]: process has died [pid 1067, exit code 1, cmd '/spot_ws/install/spot_driver/lib/spot_driver/spot_alerts --ros-args -r __node:=spot_alerts -r __ns:=/ -p use_sim_time:=False'].

and

[spot_ros2-1] [INFO] [1771893748.237147771] [spot_ros2]: Hi from spot_driver.
[spot_ros2-1] [INFO] [1771893748.273824298] [spot_ros2]: Starting ROS driver for Spot
[spot_ros2-1] [INFO] [1771893748.899415629] [spot_ros2]: Found estop!
[spot_ros2-1] [INFO] [1771893748.901132830] [spot_ros2]: Shutting down ROS driver for Spot
[spot_ros2-1] Traceback (most recent call last):
[spot_ros2-1]   File "/spot_ws/install/spot_driver/lib/spot_driver/spot_ros2", line 3182, in <module>
[spot_ros2-1]     main()
[spot_ros2-1]   File "/spot_ws/install/synchros2/lib/python3.12/site-packages/synchros2/process.py", line 190, in __call__
[spot_ros2-1]     with scope.top(argv, global_=True, interruptible=self._interruptible, **scope_kwargs) as self._scope:
[spot_ros2-1]   File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__
[spot_ros2-1]     self.gen.throw(value)
[spot_ros2-1]   File "/spot_ws/install/synchros2/lib/python3.12/site-packages/synchros2/scope.py", line 556, in top
[spot_ros2-1]     stack.close()
[spot_ros2-1]   File "/usr/lib/python3.12/contextlib.py", line 618, in close
[spot_ros2-1]     self.__exit__(None, None, None)
[spot_ros2-1]   File "/usr/lib/python3.12/contextlib.py", line 610, in __exit__                                                                                                                                       [spot_ros2-1]     raise exc_details[1]
[spot_ros2-1]   File "/usr/lib/python3.12/contextlib.py", line 595, in __exit__
[spot_ros2-1]     if cb(*exc_details):
[spot_ros2-1]        ^^^^^^^^^^^^^^^^
[spot_ros2-1]   File "/spot_ws/install/synchros2/lib/python3.12/site-packages/synchros2/scope.py", line 165, in __exit__
[spot_ros2-1]     self._stack.close()
[spot_ros2-1]   File "/usr/lib/python3.12/contextlib.py", line 618, in close
[spot_ros2-1]     self.__exit__(None, None, None)
[spot_ros2-1]   File "/usr/lib/python3.12/contextlib.py", line 610, in __exit__
[spot_ros2-1]     raise exc_details[1]
[spot_ros2-1]   File "/usr/lib/python3.12/contextlib.py", line 595, in __exit__
[spot_ros2-1]     if cb(*exc_details):
[spot_ros2-1]        ^^^^^^^^^^^^^^^^
[spot_ros2-1]   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
[spot_ros2-1]     next(self.gen)
[spot_ros2-1]   File "/spot_ws/install/synchros2/lib/python3.12/site-packages/synchros2/executors.py", line 837, in foreground
[spot_ros2-1]     if not executor.shutdown(timeout_sec=5.0):
[spot_ros2-1]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[spot_ros2-1]   File "/spot_ws/install/synchros2/lib/python3.12/site-packages/synchros2/executors.py", line 768, in shutdown
[spot_ros2-1]     AutoScalingMultiThreadedExecutor.Task(task, entity, node) for task, entity, node in self._tasks
[spot_ros2-1]                                                                                         ^^^^^^^^^^^
[spot_ros2-1] AttributeError: 'AutoScalingMultiThreadedExecutor' object has no attribute '_tasks'
[ERROR] [spot_ros2-1]: process has died [pid 1062, exit code 1, cmd '/spot_ws/install/spot_driver/lib/spot_driver/spot_ros2 --ros-args -r __node:=spot_ros2 -r __ns:=/ -p use_sim_time:=False --params-file /tmp/launc
h_params_0kk870_l --params-file /tmp/launch_params_9zs_k23z'].

Upon investigating the failure, I noticed that AutoScalingMultiThreadedExecutor's shutdown method in synchros2/executors.py file attempts to access the self._tasks attribute inherited from rclpy.executors.Executor.

If you look into the Executor's source code, jazzy and beyond implementations remove the self._tasks attribute and replace it with self._ready_tasks which is a different type.

• Humble version of Executor class

• Jazzy version of Executor class

The remainder of the spot_driver code seems fine except for this issue, and I'm not sure if there is a similar workaround that would be backward compatible between ROS 2 Humble and ROS 2 Jazzy. I'll keep looking at the code to see if there may be a work around and provide a patch.

[mhidalgo-rai]

Thanks for the bug report @zmk5! It looks like we should be paying attention to self._pending_tasks in ROS 2 Jazzy. Upstream is still leaving tasks unawaited on shutdown so we can't remove this workaround.

[zmk5]

Sorry if I am misunderstanding (or if this is a stupid question) but do you mean the synchros2 upstream or the rclpy upstream?

[zmk5]

I wrote something like this:

        if done:
            with self._spin_lock:
                # rclpy.executors.Executor base implementation leaves tasks
                # unawaited upon shutdown. Do the housekeepng.
                if hasattr(self, "_tasks"):  # ROS 2 Humble
                    known_tasks = [
                        AutoScalingMultiThreadedExecutor.Task(task, entity, node) for task, entity, node in self._tasks
                    ] + list(self._work_in_progress)
                else:  # ROS 2 Jazzy and beyond
                    known_tasks = [
                        AutoScalingMultiThreadedExecutor.Task(task, task_data.source_entity, task_data.source_node)
                        for task, task_data in self._pending_tasks.items()
                    ] + list(self._work_in_progress)
                for task in known_tasks:
                    task.cancel()

and now I get this error for both spot_alerts and spot_ros2:

[spot_ros2-1] Traceback (most recent call last):
[spot_ros2-1]   File "/spot_ws/install/spot_driver/lib/spot_driver/spot_ros2", line 3182, in <module>
[spot_ros2-1]     main()
[spot_ros2-1]   File "/spot_ws/install/synchros2/lib/python3.12/site-packages/synchros2/process.py", line 212, in __call__
[spot_ros2-1]     return func_taking_argv(rclpy.utilities.remove_ros_args(argv))
[spot_ros2-1]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[spot_ros2-1]   File "/spot_ws/install/spot_driver/lib/spot_driver/spot_ros2", line 3178, in main
[spot_ros2-1]     ros_process.spin(SpotROS)
[spot_ros2-1]   File "/spot_ws/install/synchros2/lib/python3.12/site-packages/synchros2/process.py", line 446, in spin
[spot_ros2-1]     process.spin(factory, *args, **kwargs)
[spot_ros2-1]   File "/spot_ws/install/synchros2/lib/python3.12/site-packages/synchros2/scope.py", line 485, in spin
[spot_ros2-1]     self._executor.spin()
[spot_ros2-1]   File "/opt/ros/jazzy/lib/python3.12/site-packages/rclpy/executors.py", line 359, in spin
[spot_ros2-1]     self._spin_once_impl()
[spot_ros2-1]   File "/opt/ros/jazzy/lib/python3.12/site-packages/rclpy/executors.py", line 422, in _spin_once_impl
[spot_ros2-1]     raise NotImplementedError()
[spot_ros2-1] NotImplementedError

The variable known_tasks and self._pending_tasks end up returning an empty list and dictionary, respectively. I'm not sure if that has anything to do with this new error though...

[mhidalgo-rai]

Sorry if I am misunderstanding (or if this is a stupid question) but do you mean the synchros2 upstream or the rclpy upstream?

rclpy upstream.

The variable known_tasks and self._pending_tasks end up returning an empty list and dictionary, respectively. I'm not sure if that has anything to do with this new error though...

They do not. It looks like #193 didn't cover all ground. We have an _spin_once_impl method now in Jazzy (and Kilted) that needs implementation. IIUC the solution is to rename _do_spin_once to _spin_once_impl in synchros2. That should have us covered on Humble and onwards.

[zmk5]

I just tried it and that works! Would you like me to make a pull request with these changes?

I know the hasattr() section of that code is not the cleanest, but I can do the _do_spin_once to _spin_once_impl change.

[mhidalgo-rai]

Would you like me to make a pull request with these changes?

Please do.

I know the hasattr() section of that code is not the cleanest, but I can do the _do_spin_once to _spin_once_impl change.

We need a solution for both, so a patch is appreciated 😃