[Question] How to attach a camera sensor on manager based environments?

[ssunkyu]

Hi, I'm trying to attach a camera sensor to robots in manager-based environments, but I am encountering some errors.

In my robot environment configuration file, I have the following code:

self.scene.camera: CameraCfg = CameraCfg(
    prim_path="{ENV_REGEX_NS}/Robot/base/camera",
    update_period=0.1,
    height=800,
    width=800,
    data_types=["rgb",],
    spawn=sim_utils.PinholeCameraCfg(
        focal_length=24.0, focus_distance=400.0, horizontal_aperture=20.955, clipping_range=(0.1, 1.0e5)
    ),
    offset=CameraCfg.OffsetCfg(pos=(0.510, 0.0, 0.015), rot=(0.5, -0.5, 0.5, -0.5), convention="ros"),
)

However, when I run a Python script (like lift_cube_sm.py in the example), I receive the following error message:

Traceback (most recent call last):
  File "/home/isaac/IsaacLabExtensionTemplate/scripts/lift_cube_sm.py", line 360, in <module>
    main()
  File "/home/isaac/IsaacLabExtensionTemplate/scripts/lift_cube_sm.py", line 272, in main
    env.reset()
  File "/home/isaac/miniconda3/envs/isaaclab/lib/python3.10/site-packages/gymnasium/wrappers/order_enforcing.py", line 61, in reset
    return self.env.reset(**kwargs)
  File "/home/isaac/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/envs/manager_based_env.py", line 252, in reset
    self._reset_idx(indices)
  File "/home/isaac/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/envs/manager_based_rl_env.py", line 325, in _reset_idx
    self.scene.reset(env_ids)
  File "/home/isaac/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/scene/interactive_scene.py", line 343, in reset
    sensor.reset(env_ids)
  File "/home/isaac/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/sensors/camera/camera.py", line 352, in reset
    raise RuntimeError(
RuntimeError: Camera could not be initialized. Please ensure --enable_cameras is used to enable rendering.

I added --enable_cameras when running the Python script, but it still fails. Is there something wrong with how I added the camera to the environment? The examples I found only show how to add cameras in direct environments, so I'm unsure how to do it in manager-based environments.

Thank you for your help!

[glvov-bdai]

I've recently added the first example of a manager based environment with cameras:

https://github.com/isaac-sim/IsaacLab/blob/main/source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/manager_based/classic/cartpole/cartpole_camera_env_cfg.py

I've tested the above with the RL games pipeline. You can see how I set up the camera and the observation in the example above. Please let me know if you can't get the example to work I'd be happy to assist further!

[ssunkyu]

Thanks for your response!

However, the suggested solution didn’t work for me. I was trying to add a camera in the state_machine. Here’s the file I was working with: lift_cube_sm.py.

The solution I found is as follows:

# In lift_cube_sm.py, 
# Originally: app_launcher = AppLauncher(headless=args_cli.headless)
app_launcher = AppLauncher(headless=args_cli.headless, enable_cameras=True)

Instead of adding the --enable_cameras flag when running the script (e.g., python lift_cube_sm.py --enable_cameras), I passed the argument directly in the script, which resolved the error:

RuntimeError: Camera could not be initialized. Please ensure --enable_cameras is used to enable rendering.