Training issues: RuntimeError: The MUJOCO_EGL_DEVICE_ID environment variable must be an integer between 0 and 0 (inclusive)

[Duckyee728]

I run the code without any change on swarm (in a docker environment):
bash scripts/train_rl_vllm_ray_fsdp.sh 0,1,2,3,4,5,6,7 0,1,2,3,4,5,6,7,8,9

Traceback (most recent call last):
File "/path/to/project/ppo/envs/libero_env.py", in
env_creators.append(lambda args=env_args: OffScreenRenderEnv(**args))
File "/path/to/python/site-packages/robosuite/environments/base.py", line 299, in _reset_internal
render_context = MjRenderContextOffscreen(self.sim, device_id=self.render_gpu_device_id)
File "/path/to/python/site-packages/robosuite/utils/binding_utils.py", line 78, in init
self.gl_ctx = GLContext(max_width=max_width, max_height=max_height, device_id=self.device_id)
File "/path/to/python/site-packages/robosuite/renderers/context/egl_context.py", line 121, in init
EGL_DISPLAY = create_initialized_egl_device_display(device_id=device_id)
File "/path/to/python/site-packages/robosuite/renderers/context/egl_context.py", line 61, in create_initialized_egl_device_display
raise RuntimeError(
RuntimeError: The MUJOCO_EGL_DEVICE_ID environment variable must be an integer between 0 and 0 (inclusive), got 7.

I tried to set the MUJOCO_EGL_DEVICE_ID = 0, but got annother error:

AssertionError: MUJOCO_EGL_DEVICE_ID needs to be set to one of the device id specified in CUDA_VISIBLE_DEVICES

Any suggestion will be appreciated! Thanks!

[GuanxingLu]

Thanks for your interest! The fault is the conflict between Ray (which automatically sets CUDA_VISIBLE_DEVICES in each instance to its local rank, e.g., 7) and robosuite (which requires automatically using CUDA_VISIBLE_DEVICES to render, but your device only supports rendering on the 0th GPU). You can try this to fix it:

env_args = {
    "bddl_file_name": bddl_file,
    "camera_heights": resolution,
    "camera_widths": resolution,
}
os.environ.pop("CUDA_VISIBLE_DEVICES", None) # add this
env_creators.append(lambda args=env_args: OffScreenRenderEnv(**args))