[Question] RayCaster vs Camera API inconsistency?

[delenius]

I created a Camera and a RayCaster via config class and a ManagerBasedRLEnvCfg:

from isaaclab_tasks.manager_based.locomotion.velocity.velocity_env_cfg import MySceneCfg

@configclass
class Go2SceneCfg(MySceneCfg):
    camera = CameraCfg(
        prim_path="{ENV_REGEX_NS}/Robot/base/front_cam",
        update_period=0.1,
        height=480,
        width=640,
        data_types=["rgb", "depth"],
        spawn=sim_utils.PinholeCameraCfg(
            focal_length=24.0, focus_distance=400.0, horizontal_aperture=20.955, clipping_range=(0.1, 1.0e5)
        ),
        offset=CameraCfg.OffsetCfg(pos=(0.32487, -0.00095, 0.05362), rot=(0.5, -0.5, 0.5, -0.5), convention="ros"),
    )

    ray_caster = RayCasterCfg(
        prim_path="{ENV_REGEX_NS}/Robot/base/lidar_cage",
        update_period=1 / 60,
        offset=RayCasterCfg.OffsetCfg(pos=(0, 0, 0.5)),
        mesh_prim_paths=["/World/ground"],
        attach_yaw_only=True,
        pattern_cfg=patterns.LidarPatternCfg(
            channels=100, vertical_fov_range=[-90, 90], horizontal_fov_range=[-90, 90], horizontal_res=1.0
        ),
        debug_vis=False,
    )

The camera can be created fine in this way, but the RayCaster causes an error,

Traceback (most recent call last):
  File "/homes/elenius/repos/IsaacLab/source/isaaclab/isaaclab/sensors/sensor_base.py", line 66, in <lambda>
    lambda event, obj=weakref.proxy(self): obj._initialize_callback(event),
  File "/homes/elenius/repos/IsaacLab/source/isaaclab/isaaclab/sensors/sensor_base.py", line 273, in _initialize_callback
    self._initialize_impl()
  File "/homes/elenius/repos/IsaacLab/source/isaaclab/isaaclab/sensors/ray_caster/ray_caster.py", line 130, in _initialize_impl
    raise RuntimeError(f"Failed to find a prim at path expression: {self.cfg.prim_path}")
RuntimeError: Failed to find a prim at path expression: /World/envs/env_.*/Robot/base/lidar_cage

Apparently, for RayCaster, but not for Camera, the prim has to be created first.
We can also see this in the source code for Camera,

        self._view = XFormPrim(self.cfg.prim_path, reset_xform_properties=False)

vs the source code for RayCaster, where it checks for existence of the prim:

        prim = sim_utils.find_first_matching_prim(self.cfg.prim_path)
        if prim is None:
            raise RuntimeError(f"Failed to find a prim at path expression: {self.cfg.prim_path}")
        # create view based on the type of prim
        if prim.HasAPI(UsdPhysics.ArticulationRootAPI):
            self._view = self._physics_sim_view.create_articulation_view(self.cfg.prim_path.replace(".*", "*"))
            found_supported_prim_class = True
        elif prim.HasAPI(UsdPhysics.RigidBodyAPI):
            self._view = self._physics_sim_view.create_rigid_body_view(self.cfg.prim_path.replace(".*", "*"))
            found_supported_prim_class = True
        else:
            self._view = XFormPrim(self.cfg.prim_path, reset_xform_properties=False)
            found_supported_prim_class = True
            omni.log.warn(f"The prim at path {prim.GetPath().pathString} is not a physics prim! Using XFormPrim.")
        # check if prim view class is found
        if not found_supported_prim_class:
            raise RuntimeError(f"Failed to find a valid prim view class for the prim paths: {self.cfg.prim_path}")

Could we have RayCaster behave the same as Camera here (and just have it create a XFormPrim if none exists)?

[RandomOakForest]

Thanks for posting this, a very interesting discussion starter. I'll move this into our Discussions section for the team to follow up.

[delenius]

Looking at it some more, I think the real issue is that RayCasterCfg has no spawn field (or SpawnCfg). CameraCfg has this, which is why the camera prim is spawned for me automatically, but not the ray caster one:

    spawn: PinholeCameraCfg | FisheyeCameraCfg | None = MISSING
    """Spawn configuration for the asset.

    If None, then the prim is not spawned by the asset. Instead, it is assumed that the
    asset is already present in the scene.
    """

Shouldn't RayCasterCfg also have this? More generally, I am trying to add a lidar to the unitree go2 model. Any suggestions welcome. Would be nice if I didn't have to modify the robot model to do this. Note that I cannot add the prim in my code before I initialize the environment from the config classes, because at that point there is no robot yet.

As a workaround, I was able to use prim_path="{ENV_REGEX_NS}/Robot/base" directly, an approach I also see in the IsaacLab source, e.g. in velocity_env_cfg.py.

[pascal-roth]

So the raycaster is a virtual sensor; it does not require its own prim, in contrast to the used camera (and therefore also doesn't need a spawn function). It only requires a prim to get its position (faster with articulation or rigid body, slower with form). To make the computation as fast as possible, it is recommended to be attached to the base frame of your robot.

The solution which you see in IsaacLab source is therefore not a workaround but the intended use case