Support of Resnet encoder

[shahizat]

Hello,

I just wanted to know, does rsl_rl support a ResNet encoder? I want to pass output from a camera on Nvidia IsaacLab, image and depth data, as the observation input to train a navigation task for a wheeled robot.

Any guidance on how to do this would be appreciated.

Thanks.

[ClemensSchwarke]

Hi @shahizat, this should be pretty simple! Just modify actor_critic.py to your desired architecture.

[shahizat]

ok @ClemensSchwarke , thanks! Sorry for the newbie question, I understand that Isaac Lab already provides utilities to encode images into compact feature vectors, which can then be concatenated with other state observations and fed into the MLP in the existing ActorCritic without modification. Just to clarify, what’s the advantage of using pre-extracted ResNet features from Isaac Lab (like image_features) instead of implementing a custom network directly? What is recommended way to proceed for the training of robot for obstacle avoidance task?

[ClemensSchwarke]

Hey, not sure I fully understand your question. First, if you work with images it is not a good idea to flatten them and feed them through an mlp. If you want to use a resnet as an image encoder, you can pass the sensor images directly to rsl_rl as an observation group (preserving the shape) and then add the resnet in your custom actor_critic. Not sure if this answers your question, feel free to ask again if it is not clear.

[shahizat]

@ClemensSchwarke, I think I understand what you mean, but I just need an example. This one seems good:https://github.com/isaac-sim/IsaacLab/pull/2027/files . Are there any plans to implement something like that?

[ClemensSchwarke]

Feel free to have a look at that, but many things got simpler with the new observation handling of rsl_rl. Maybe you can have a look at this PR: #114. You just need to add a resnet instead of the CNN.