Adds Raycaster with tracking for dynamic meshes

[renezurbruegg]

Description

This PR introduces MultiMeshRayCaster and MultiMeshRayCasterCamera, an extension of the default RayCaster with the following enhancements:

1. Raycasting against multiple target types : Supports primitive shapes (spheres, cubes, …) as well as arbitrary meshes.
2. Dynamic mesh tracking : Keeps track of specified meshes, enabling raycasting against moving parts (e.g., robot links, articulated bodies, or dynamic obstacles).
3. Memory-efficient caching : Avoids redundant memory usage by caching and reusing duplicate meshes.

This is joint work with @pascal-roth and @Mayankm96.

The default RayCaster was limited to static environments and required manual handling of moving meshes, which restricted its use for robotics scenarios where robots or obstacles move dynamically.

MultiMeshRayCaster addresses these limitations by and now supports raycasting against robot parts and other moving entities.

---

Usage

For a quick demo, run:

python scripts/demos/sensors/multi_mesh_raycaster.py --num_envs 16 --asset_type <allegro_hand|anymal_d|multi>

Drop-in replacement

Example change to migrate from RayCasterCfg to MultiMeshRayCasterCfg:

- ray_caster_cfg = RayCasterCfg(
+ ray_caster_cfg = MultiMeshRayCasterCfg(
      prim_path="{ENV_REGEX_NS}/Robot",
      mesh_prim_paths=[
         "/World/Ground",
+         MultiMeshRayCasterCfg.RaycastTargetCfg(target_prim_expr="{ENV_REGEX_NS}/Robot/LF_.*/visuals"),
+         MultiMeshRayCasterCfg.RaycastTargetCfg(target_prim_expr="{ENV_REGEX_NS}/Robot/RF_.*/visuals"),
+         MultiMeshRayCasterCfg.RaycastTargetCfg(target_prim_expr="{ENV_REGEX_NS}/Robot/LH_.*/visuals"),
+         MultiMeshRayCasterCfg.RaycastTargetCfg(target_prim_expr="{ENV_REGEX_NS}/Robot/RH_.*/visuals"),
+         MultiMeshRayCasterCfg.RaycastTargetCfg(target_prim_expr="{ENV_REGEX_NS}/Robot/base/visuals"),
      ],
      pattern_cfg=patterns.GridPatternCfg(resolution=resolution, size=(5.0, 5.0)),
 )

---

Benchmarking & Validation

To benchmark the new raycaster, run:

python scripts/benchmarks/benchmark_ray_caster.py

Then plot the results with:

python scripts/benchmarks/plot_raycast_results.py

This will generate outputs under:
outputs/benchmarks/raycast_benchmark...

Example plots

---

Type of Change

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Checklist

• I have run the pre-commit checks with ./isaaclab.sh --format
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have updated the changelog and the corresponding version in the extension's config/extension.toml file
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[greptile-apps]

Greptile Overview

Greptile Summary

This PR adds MultiMeshRayCaster and MultiMeshRayCasterCamera, extending the existing raycaster to support dynamic mesh tracking, multiple target types, and memory-efficient mesh caching.

Key Changes:

• New MultiMeshRayCaster class enabling raycasting against moving meshes (robot links, articulated bodies) with per-environment tracking
• Warp kernel infrastructure (raycast_static_meshes_kernel, raycast_dynamic_meshes_kernel) for GPU-accelerated raycasting with atomic operations to handle multiple meshes per ray
• Mesh deduplication system comparing vertex arrays to avoid redundant GPU memory usage across environments
• Camera variant (MultiMeshRayCasterCamera) generating depth images with proper coordinate frame transformations
• New isaaclab.utils.mesh module converting USD primitives (spheres, cubes, cylinders, etc.) to trimesh objects
• Comprehensive test coverage (1114 new test lines) validating multi-mesh scenarios, dynamic transforms, and mesh ID tracking

Technical Implementation:

• Class-level mesh dictionaries (meshes, mesh_offsets, mesh_views) shared across instances for memory efficiency
• Per-environment mesh ID arrays (mesh_ids_wp) enabling batched raycasting across multiple environments
• Optional transform tracking via track_mesh_transforms flag - static meshes skip pose updates for performance
• Mesh merging capability combining multiple child meshes into single raycasting target

Issues Found:

• Minor: Empty mesh prim handling at line 206 uses continue which could leave _num_meshes_per_env incomplete, potentially causing KeyError in later processing
• Most previously reported issues are either false positives or acknowledged design choices (e.g., Warp atomic_min race condition is documented upstream issue)

Confidence Score: 4/5

• Safe to merge with one minor logic issue noted that should be reviewed but likely doesn't affect typical usage scenarios
• The implementation is well-architected with comprehensive test coverage (1114 new test lines), proper GPU memory management through mesh deduplication, and clean separation of concerns. The warp kernel race condition is a known upstream issue documented in the code. The one issue identified (incomplete _num_meshes_per_env handling when mesh prims are empty) is edge-case behavior that likely won't affect normal operation but should be validated. The PR successfully achieves its goal of enabling dynamic mesh raycasting while maintaining backward compatibility.
• source/isaaclab/isaaclab/sensors/ray_caster/multi_mesh_ray_caster.py - validate empty mesh prim handling around line 206-215 doesn't cause downstream KeyErrors

Important Files Changed

File Analysis

Filename	Score	Overview
source/isaaclab/isaaclab/sensors/ray_caster/multi_mesh_ray_caster.py	4/5	Core implementation of multi-mesh raycasting with dynamic tracking, mesh deduplication, and per-environment mesh management. Minor issues with early-return logic and mesh counting.
source/isaaclab/isaaclab/utils/warp/kernels.py	4/5	New warp kernels for static and dynamic mesh raycasting. Race condition is documented as known issue with Warp library. Docstring shape mismatches noted in previous reviews.
source/isaaclab/isaaclab/utils/warp/ops.py	4/5	Added raycast_single_mesh and raycast_dynamic_meshes functions with proper tensor management. Quaternion conversion and initialization patterns are correct.
source/isaaclab/isaaclab/sensors/ray_caster/multi_mesh_ray_caster_camera.py	4/5	Camera implementation with image-based raycasting output. Properly handles depth clipping and coordinate frame transformations. Multiple inheritance from RayCasterCamera and MultiMeshRayCaster works as expected.
source/isaaclab/isaaclab/utils/mesh.py	5/5	New utility module for converting USD geometric primitives to trimesh objects. Clean implementation with proper axis handling for cylinders and capsules.
source/isaaclab/isaaclab/sensors/ray_caster/ray_caster.py	4/5	Modified base RayCaster class to support mesh tracking infrastructure. Added class-level mesh dictionaries and tracking logic.

Sequence Diagram

sequenceDiagram
    participant User
    participant MultiMeshRayCaster
    participant MultiMeshRayCasterCamera
    participant WarpOps
    participant WarpKernels
    participant MeshUtils
    
    Note over User,MeshUtils: Initialization Phase
    User->>MultiMeshRayCaster: __init__(cfg)
    MultiMeshRayCaster->>MultiMeshRayCaster: _initialize_warp_meshes()
    MultiMeshRayCaster->>MeshUtils: find_matching_prims(target_prim_expr)
    MeshUtils-->>MultiMeshRayCaster: target_prims[]
    
    loop For each target prim
        MultiMeshRayCaster->>MeshUtils: get_all_matching_child_prims()
        MeshUtils-->>MultiMeshRayCaster: mesh_prims[]
        MultiMeshRayCaster->>MeshUtils: create_trimesh_from_geom_*()
        MeshUtils-->>MultiMeshRayCaster: trimesh_mesh
        MultiMeshRayCaster->>MultiMeshRayCaster: Check for duplicate vertices
        alt Duplicate found and reference_meshes=True
            MultiMeshRayCaster->>MultiMeshRayCaster: Reuse existing wp_mesh.id
        else New mesh
            MultiMeshRayCaster->>WarpOps: convert_to_warp_mesh()
            WarpOps-->>MultiMeshRayCaster: wp_mesh
        end
        MultiMeshRayCaster->>MultiMeshRayCaster: Store in class-level meshes dict
    end
    
    MultiMeshRayCaster->>MultiMeshRayCaster: Create mesh_ids_wp array (num_envs x num_meshes)
    alt track_mesh_transforms=True
        MultiMeshRayCaster->>MultiMeshRayCaster: _get_trackable_prim_view()
        MultiMeshRayCaster->>MultiMeshRayCaster: Store in mesh_views dict
    end
    
    Note over User,WarpKernels: Update/Raycast Phase
    User->>MultiMeshRayCaster: data (property access)
    MultiMeshRayCaster->>MultiMeshRayCaster: _update_outdated_buffers()
    MultiMeshRayCaster->>MultiMeshRayCaster: _update_buffers_impl(env_ids)
    
    alt track_mesh_transforms=True
        loop For each tracked mesh view
            MultiMeshRayCaster->>MultiMeshRayCaster: obtain_world_pose_from_view()
            MultiMeshRayCaster->>MultiMeshRayCaster: Apply mesh_offsets
            MultiMeshRayCaster->>MultiMeshRayCaster: Update _mesh_positions_w, _mesh_orientations_w
        end
    end
    
    MultiMeshRayCaster->>WarpOps: raycast_dynamic_meshes()
    WarpOps->>WarpOps: Initialize output tensors
    WarpOps->>WarpOps: Convert quaternions wxyz->xyzw
    
    alt Static meshes (no transforms)
        WarpOps->>WarpKernels: raycast_static_meshes_kernel()
    else Dynamic meshes (with transforms)
        WarpOps->>WarpKernels: raycast_dynamic_meshes_kernel()
    end
    
    WarpKernels->>WarpKernels: Thread per (mesh_id, env, ray)
    loop For each mesh
        WarpKernels->>WarpKernels: Transform ray to mesh local space
        WarpKernels->>WarpKernels: wp.mesh_query_ray()
        alt Hit found and closer than current
            WarpKernels->>WarpKernels: atomic_min(ray_distance)
            WarpKernels->>WarpKernels: Update ray_hits, ray_normal, ray_face_id, ray_mesh_id
        end
    end
    
    WarpKernels-->>WarpOps: Hit results
    WarpOps-->>MultiMeshRayCaster: ray_hits_w, mesh_ids
    MultiMeshRayCaster->>MultiMeshRayCaster: Store in _data container
    MultiMeshRayCaster-->>User: MultiMeshRayCasterData
    
    Note over User,WarpKernels: Camera-Specific Flow
    User->>MultiMeshRayCasterCamera: data (property access)
    MultiMeshRayCasterCamera->>MultiMeshRayCasterCamera: _update_buffers_impl()
    MultiMeshRayCasterCamera->>MultiMeshRayCasterCamera: Update mesh transforms (inherited)
    MultiMeshRayCasterCamera->>WarpOps: raycast_dynamic_meshes()
    WarpOps-->>MultiMeshRayCasterCamera: ray_hits, ray_depth, ray_normal, mesh_ids
    
    alt distance_to_image_plane requested
        MultiMeshRayCasterCamera->>MultiMeshRayCasterCamera: Transform to camera frame
        MultiMeshRayCasterCamera->>MultiMeshRayCasterCamera: Extract z-component
        MultiMeshRayCasterCamera->>MultiMeshRayCasterCamera: Apply depth clipping
    end
    
    MultiMeshRayCasterCamera->>MultiMeshRayCasterCamera: Reshape to image shape (H, W)
    MultiMeshRayCasterCamera-->>User: MultiMeshRayCasterCameraData

Loading

[greptile-apps]

_{22 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

logic: continue on empty mesh prims skips the target but allows processing to continue with potentially incomplete mesh setup. If no mesh prims are found for any target in this loop, the code continues but _num_meshes_per_env won't be updated correctly for this target. Later code at line 310 expects all target_cfgs to have entries in _num_meshes_per_env, which could cause KeyError. Should this be tracked or validated at the end of mesh initialization?

[XiaoYi-Wei]

Simulation startup becomes extremely slow when using the RayCaster sensor with large numbers of environments

Description

When running the RayCaster sensor branch in IsaacSim, the simulation initialization time increases drastically as the number of environments grows.

The program does not crash, but the simulation hangs for a long time at the message:
[INFO]: Starting the simulation. This may take a few seconds. Please wait...

---

Steps to Reproduce

1. Launch the simulation using the RayCaster sensor branch.
2. Set the number of environments to a large value (e.g., 1000, 2000, or more).
3. Observe the time it takes to start the simulation.

---

Observed Behavior

• The console shows normal terrain and scene creation messages, then remains at picture

• The program stays in this state for a long time without proceeding.

• Startup time increases roughly linearly with the number of environments.

Number of Environments	Approx. Startup Time
1	Immediate
1000	~1 minute
4000	Several minutes

---

Expected Behavior

The simulation should start within a reasonable time regardless of the number of environments, consistent with other sensors or configurations.

---

Environment

• IsaacSim version: [your version]
• RayCaster branch: [branch name or commit hash]
• GPU / OS: [your specs]
• Number of environments tested: up to 4000

[XiaoYi-Wei]

### Issue Description

I found a potential bug in the file:

`source/isaaclab/isaaclab/sensors/ray_caster/multi_mesh_ray_caster_camera.py`  
at **line 178**.

The current code is:

```python
self.ray_hits_w, ray_depth, ray_normal, _, ray_mesh_ids = raycast_dynamic_meshes()

However, the first returned value of raycast_dynamic_meshes() has the shape:

[len(env_ids), num_rays, 3]

This causes a CUDA kernel out-of-bounds access, because self.ray_hits_w expects indexing over all environments, while the returned tensor only corresponds to len(env_ids).
The issue doesn't throw an error immediately because self.ray_hits_w is not used later in this path, but the kernel still performs invalid memory access.

Suggested Fix

It seems the assignment should correctly write only to the selected environments:

self.ray_hits_w[env_ids], ray_depth, ray_normal, _, ray_mesh_ids = raycast_dynamic_meshes()

This aligns the shapes and prevents hidden OOB kernel access.

[pascal-roth]

@XiaoYi-Wei thanks for the catch, should be fixed now

[pascal-roth]

Should have addressed everything that was still open.

TODO: rerun the examples!

[pascal-roth]

need to merge #3924 first

[kellyguo11]

@pascal-roth @renezurbruegg could someone help take a look at the doc build failure?

[Mayankm96]

@kellyguo11 I made an MR now to address the comments remaining here. I think Pascal and Rene are OoO now.