RTX Lidar: How to process Intensity ; Please provide a tutorial / example...

[BenjaminYde]

Problem

Problem statement: How do we create an RTX lidar that can publish intensity data via ROS2.

What I want

Currently there is no clear documentation / tutorial / example on how to create an RTX lidar that can publish intensity data.

What is the recommended way of doing this?

• Creating a specialized material?
• Adding a specialized USD schema to a prim so we can set the property?
• Is it possible to do this in the isaac sim editor? Or can this only be done via python scripting?

We want to know how we can do this in 2 ways:

• Via isaac sim editor & USD:
  • How can we set this up using only the isaac sim editor & USD without using any python.
  • In the editor, is it possible to see the properties of certain objects that have a different intensity?
• Via scripting (python) :
  • How can we set this up using python, for example when we create a simulation.py script that creates a headless simulation.

Tutorial / example

The example can be very simple:

• Create USD scene with 3 stacked cubes where 1 of the cubes has a different intensity
• Create 1 pre-defined RTX Lidar (such as an ouster OS0)
• Create a ROS2 publisher that sends the the 3D position data but also the intensity (of all points).
• Visualize the ROS2 pointcloud in rviz2 + show the intensity data.

What I found on the documentation:

• nvidia docs: rtx senso annotators
  • On the docs you talk about the IsaacExtractRTXSensorPointCloud but how do you use this?
• nvidia docs: Rtx sensor visual materials
• nvidia docs: GenericModelOutput
• github: isaac-sim: api examples isaacsim.sensors.rtx

Tip

• Maybe create a video about it that you can post on NVIDIA Omnivese channel during the Robotics Office Hours. I'm sure a lot of people would want to use this!
• Adding screenshots to the documentation of IsaacSim always helps!

Reference to Nvidia Forum post:

I asked for this 9 jan 2025 for isaac sim 4.2.0 but still no results. This is something that is requested a LOT by the community.
See the Nvidia forum post RTX Lidar PointCloud 3D: How to set a custom intensity on objects? Changing the material properties have no effect
for more information.

Another post since Juli 01 2024 about the same topic here

Also in the Discord, nobody seems to be able to give me an example. (see here)

[vick-yu]

Thank you for raising this important issue and for the detailed problem description. We will check with our team on how to address this and get back to you with an update as soon as possible.

[lesurJ]

@benjaminde I don't think it is possible to publish directly the intensities values to ROS2 with the current version of the simulator.
However, i have found a trick to still retrieve the intensities from the python code with an annotator. This works for Isaac Sim 5.0.0 and i did not try for previous releases.

If i understand correctly, Isaac Sim is using annotators to enrich the point cloud created by the Lidars. As of today, only three annotators are available for RTXLidar but only one supports outputting the intensities: IsaacCreateRTXLidarScanBuffer. So we have to use this annotator and find a way to pass it the "outputIntensity" flag described in the documentation (and others flags if needed).

Now we take a look at the registration of the annotators happening in this file <isaacsim-root-path>/exts/isaacsim.sensors.rtx/isaacsim/sensors/rtx/impl/extension.py at line 141. We can see that the registration of IsaacCreateRTXLidarScanBuffer (line 154) is NOT passing any of the flags that we need and thus they are not available in the output. In comparison, the IsaacCreateRTXLidarScanBufferForFlatscan (line 169) is passing the args through the init_params argument. So the default behavior of our Buffer annotator is to only have the basic xyz data. So we could modify the registration of the annotator to accept outputting intensity. However, i would not recommend it.

To create the RTX Lidar, I used the python API. This LidarRtx class (see the code at <isaacsim-root-path>/exts/isaacsim.sensors.rtx/isaacsim/sensors/rtx/impl/lidar_rtx.py) can attach the aforementioned annotators via its attach_annotator method but it is still not passing the flag that we need.
But if you take a look at the signature of the function called on line 133 (rep.AnnotatorRegistry.get_annotator()), you can see that the function accepts the init_params dict that we need!

So you can derive this LidarRtx class and override the attach_annotator method on line 211 with this:

def attach_annotator(
        self,
        annotator_name: Literal[
            "IsaacComputeRTXLidarFlatScan",
            "IsaacExtractRTXSensorPointCloudNoAccumulator",
            "IsaacCreateRTXLidarScanBuffer",
        ],
    ) -> None:
        """Attach an annotator to the Lidar sensor.

        Args:
            param annotator_name (Literal): Name of the annotator to attach. Must be one of:
                - "IsaacComputeRTXLidarFlatScan"
                - "IsaacExtractRTXSensorPointCloudNoAccumulator"
                - "IsaacCreateRTXLidarScanBuffer"
        """
        if annotator_name in self._annotators:
            carb.log_warn(
                f"Annotator {annotator_name} already attached to {self._render_product_path}"
            )
            return

        #annotator = rep.AnnotatorRegistry.get_annotator(annotator_name)
        annotator = rep.AnnotatorRegistry.get_annotator(
            annotator_name,
            init_params={
                # "outputAzimuth": True,
                # "outputElevation": True
                "outputIntensity": True, 
            },
        )
        annotator.attach([self._render_product_path])
        self._annotators[annotator_name] = annotator
        return

and to retrieve you xyz + intensity data, just to a mylidar.get_current_frame() where mylidar is the lidar object created with the python API.

Last, to publish the data to ROS2, it would have to be done by hand (and NOT via the builtin PointCloud publisher as it does not accept intensity). I can confirm i was able to successfully publish and retrieve intensity values via ROS2:

But the points are not correctly sorted along azimuth and elevation.

[neporez]

@lesurJ Thank you for sharing. Could you please share how to create the PointCloud publisher you mentioned at the end? I understand that it needs to be created through the Python API, but I’m not sure how to execute it.