[Question] How to implement functions in Isaac Lab that wrap warp kernels?

[jlowb]

This is quite a convoluted question with many nuances so I will try to be detailed to avoid back and forth.

I am working on dextrous grasping in Isaac Lab (by the way, any idea of when GR00T-Dexterity will release?); I am implementing a model that learns grasps based on point cloud input. The idea is to do pick object from dataset -> randomize properties on startup -> compute point cloud (1) -> randomize position/pose on restart -> retarget point cloud for new position/pose (2). The key is that (1) and (2) are implemented as warp kernels similar to this because surrounding the object in cameras (or otherwise) to compute the point cloud is not viable.

In essence, you can think of my situation as attempting to extend inhand_env_cfg with a few more observations/events, except the underlying funcs for the new terms are supposed to call warp kernels instead of directly computing something from the params. The problem is that there do not seem to be example environments that perform this lab/warp mixing (besides the velocity task with the TerrainGenerator/etc., but that is very abstracted), and so I am not quite sure how to compose my workflow within Isaac Lab in such a way that the data is global/persistent and can be passed to/from the kernels and the remainder of the code.

I think the approach is to create a surface_points variable of type SurfacePoints (which has attributes _mesh of type wp.Mesh and points of type torch.Tensor, and methods _extract_mesh_data, compute_surface_points which update the attributes when they are called) in InteractiveSceneCfg and then have EventCfg, ObservationsCfg, ActionsCfg modify/use the SceneEntity surface_points via funcs which call the methods. Will this work, or does SurfacePoints have to be set up as a config so that it duplicates correctly across the envs (which will have different underlying objects, so the point cloud is unique for each env), or is the "underspecified object" enough? Is there a better way to do this? Will this be persistent? (If the point clouds are to be used purely as an observation, we can extend ObservationsCfg, but I am trying to have the point clouds be a kind of global variable that can be used by all the manager terms if needed). The alternatives I see are extending SensorBase/ManagerBase/etc.

[RandomOakForest]

Thank you for posting this. You will have to implement this following this example. Add to ObservationsCfg and EventCfg. See how the ObsTerm is used in this script.