[Bug Report] Unitree Go2 Training Script Error on Rough Env

[rajivswamy]

Describe the bug

I'm experiencing some training instabilities when training the default Rough locomotion policy for the Unitree Go2. Instead of Nan values mentioned in #1999. However, this time the error is saying that the values of the std parameter in the sampling should be non-negative.

Steps to reproduce

Run the following bash script from the IsaacLab root folder:

SCRIPT_PATH="scripts/reinforcement_learning/rsl_rl/train.py"
TASK="Isaac-Velocity-Rough-Unitree-Go2-v0"
NUM_ENVS="4096"
MAX_ITERATIONS="20000"
SEED="42"

# Use this to toggle which cuda device gets used for training
export CUDA_VISIBLE_DEVICES=1
export HYDRA_FULL_ERROR=1


# Run the command with specified arguments
./isaaclab.sh -p "$SCRIPT_PATH" --task "$TASK" --num_envs "$NUM_ENVS" --seed "$SEED" --headless --max_iterations "$MAX_ITERATIONS"

Terminal Logs for Error:

Traceback (most recent call last):
  File "/home/pontryagin/rajiv/isaac_personal_march/IsaacLab/scripts/reinforcement_learning/rsl_rl/train.py", line 154, in <module>
    main()
  File "/home/pontryagin/rajiv/isaac_personal_march/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/utils/hydra.py", line 104, in wrapper
    hydra_main()
  File "/home/pontryagin/micromamba/envs/isaaclab_mar/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/home/pontryagin/micromamba/envs/isaaclab_mar/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/home/pontryagin/micromamba/envs/isaaclab_mar/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/home/pontryagin/micromamba/envs/isaaclab_mar/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/home/pontryagin/micromamba/envs/isaaclab_mar/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/home/pontryagin/micromamba/envs/isaaclab_mar/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/home/pontryagin/micromamba/envs/isaaclab_mar/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/home/pontryagin/micromamba/envs/isaaclab_mar/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/home/pontryagin/micromamba/envs/isaaclab_mar/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/pontryagin/rajiv/isaac_personal_march/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/utils/hydra.py", line 101, in hydra_main
    func(env_cfg, agent_cfg, *args, **kwargs)
  File "/home/pontryagin/rajiv/isaac_personal_march/IsaacLab/scripts/reinforcement_learning/rsl_rl/train.py", line 146, in main
    runner.learn(num_learning_iterations=agent_cfg.max_iterations, init_at_random_ep_len=True)
  File "/home/pontryagin/micromamba/envs/isaaclab_mar/lib/python3.10/site-packages/rsl_rl/runners/on_policy_runner.py", line 208, in learn
    mean_value_loss, mean_surrogate_loss, mean_entropy, mean_rnd_loss, mean_symmetry_loss = self.alg.update()
  File "/home/pontryagin/micromamba/envs/isaaclab_mar/lib/python3.10/site-packages/rsl_rl/algorithms/ppo.py", line 251, in update
    self.actor_critic.act(obs_batch, masks=masks_batch, hidden_states=hid_states_batch[0])
  File "/home/pontryagin/micromamba/envs/isaaclab_mar/lib/python3.10/site-packages/rsl_rl/modules/actor_critic.py", line 126, in act
    return self.distribution.sample()
  File "/home/pontryagin/micromamba/envs/isaaclab_mar/lib/python3.10/site-packages/torch/distributions/normal.py", line 73, in sample
    return torch.normal(self.loc.expand(shape), self.scale.expand(shape))
RuntimeError: normal expects all elements of std >= 0.0

-->

System Info

Describe the characteristic of your environment:

• Commit: 46cbb5d
• Isaac Sim Version: latest
• OS: Ubuntu 22.04
• GPU: NVIDIA RTX 6000
• CUDA: 12.4
• GPU Driver: 550.127.05

Additional context

Add any other context about the problem here.

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

• Criteria 1
• Criteria 2

[RandomOakForest]

Thank you for posting this. Could you please add a title by replacing "Bug title" in this topic after [Bug Report] in the headline? Thanks.

[legnAray]

I fix this bug by modify the actor_critic.py in rsl_rl.

I use softplus [ = log(1 + exp(x)) ] to force std>0.

    def update_distribution(self, observations):
        # compute mean
        mean = self.actor(observations)
        # compute standard deviation
        if self.noise_std_type == "scalar":
            current_std_val = nn.functional.softplus(self.std)
            std = current_std_val.expand_as(mean)
        elif self.noise_std_type == "log":
            std = torch.exp(self.log_std).expand_as(mean)
        else:
            raise ValueError(f"Unknown standard deviation type: {self.noise_std_type}. Should be 'scalar' or 'log'")
        # create distribution
        self.distribution = Normal(mean, std)

[Nate711]

I had the same issue and switched to log mode

[Nate711]

Actually even with using log mode I still get the same error that std becomes negative. Maybe it just becomes too small.