[Question] How to Access Ground Truth Environment Geometry in IsaacLab

[eugene0429]

How to Access Ground Truth Environment Geometry in IsaacLab

Hi NVIDIA Team,

I'm currently working on a deep learning project using IsaacLab, and I want to generate training data for a scene reconstruction model. The idea is to use sensor data (RGB, depth, LiDAR) from a robot (e.g., ANYmal-C) as input, and train a model to reconstruct the surrounding terrain or environment in 3D.

✅ My Setup:

• IsaacLab with TerrainImporterCfg to procedurally generate terrains (e.g., rough terrain).
• A robot equipped with multiple cameras or raycasters.
• I want to pair sensor data with ground-truth 3D geometry from the environment.

---

❓ My Key Question:

What is the proper or recommended way to directly access the terrain/environment geometry (i.e., full 3D point cloud or mesh) for ground truth labeling?

---

📌 Specifically:

• If terrain is spawned via TerrainImporterCfg, is it expected to be a Mesh prim under a known path like /World/ground?
• Are there tools/utilities in IsaacLab or Isaac Sim to extract the mesh vertices from the terrain (in world coordinates)?
• If the terrain or other objects are not Mesh types, is there a recommended way to convert or sample them for GT data?

---

👀 What I've Tried:

• Traversing the USD stage to find Mesh prims.
• Using UsdGeom.Mesh(...).GetPointsAttr().Get() and transforming to world coordinates.
• Found that many scene objects are Xform or Cone types, which are not directly sampleable.
• Sometimes terrain prims don’t appear at all in the stage (e.g., /World/ground is missing).

---

🎯 My Goal:

For every simulation frame, generate a pair:

• Input: Sensor data from the robot (e.g., RGB, depth).
• Ground Truth: Accurate 3D point cloud of the environment near the robot.

What’s the best practice for this in IsaacLab?

Thank you so much in advance — any pointers or sample code would be appreciated!

[RandomOakForest]

Thank you for posting this. For ground truth labeling you have to add a sensor to the scene, such as a camera, and use Replicator to request ground truth as needed. This doc may be a good place to start.

[Marsak24]

Hello, @RandomOakForest @eugene0429
This post is very close to what I’m trying to do. But I’m a bit confused, you referred them to the Replicator in Isaac Sim. So, is collecting data always an Isaac Sim thing? Also, does MobilityGen replace Replicator, or am I misunderstanding that?
One more thing, I wanted to follow up on the terrain generation part. If we’re using Isaac Lab for terrain generation, why can’t we just keep the entire flow in Isaac Lab? Why would we need Replicator at all? (Maybe I’m missing a concept here.)