[Question] How to use Custom CNN for the navigation task with obstacles.

[shahizat]

Hello, can the features extracted from a pre-trained ResNet18 below be used for a navigation task that involves obstacle avoidance? can I apply that same feature extraction function to process depth data as well? Thanks.

        image = ObsTerm(
            func=mdp.image_features,
            params={"sensor_cfg": SceneEntityCfg("tiled_camera"), "data_type": "rgb", "model_name": "resnet50"},
        )

[garylvov]

It's hard to tell if ResNet will be effective for a navigation task without testing, but I think it could work.

I believe the image features are compatible with the depth data, but truthfully I haven't tried it.

I would give it a shot! If the feature extraction doesn't work with depth data I'd be happy to help on a PR to fix it

[shahizat]

Hi @garylvov, Thanks for your response! I’m having a bit of trouble training the policy for my two-wheeled robot using ResNet for navigation in an obstacle environment. I’m using SKRL. Would it be better to use a CNN instead of a regular MLP for this task?

[garylvov]

Yes the resnet is frozen while the CNN can be trained specifically for your task

[shahizat]

hi @garylvov do you have any examples for mobile robots (like the JetBot or TurtleBot) that use an RGB or depth features for training via a manager-based workflow?

[garylvov]

I'd check out #3467 and leggedrobotics/rsl_rl#114

[PierrePeng]

Also, RLRover may help.

[garylvov]

That project is cool thanks for sharing!!!