[Bug Report] RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch)

[zhanghui-china]

./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Velocity-Rough-Anymal-C-v0 --headless

Log：

(issaacsim311) zhanghui@gx10-8e22:/home1/zhanghui/IsaacLab$ ./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Velocity-Rough-Anymal-C-v0 --headless 
[INFO] Using python from: /home/zhanghui/anaconda3/envs/issaacsim311/bin/python                                                               
[INFO][AppLauncher]: Using device: cuda:0
[INFO][AppLauncher]: Loading experience file: /home1/zhanghui/IsaacLab/apps/isaaclab.python.headless.kit
Loading user config located at: '/home/zhanghui/.cache/packman/chk/kit-kernel/107.3.3+isaac.229672.69cbf6ad.gl.manylinux_2_35_aarch64.release/data/Kit/Isaac-Sim/5.1/user.config.json'
[Info] [carb] Logging to file: /home/zhanghui/.cache/packman/chk/kit-kernel/107.3.3+isaac.229672.69cbf6ad.gl.manylinux_2_35_aarch64.release/logs/Kit/Isaac-Sim/5.1/kit_20260506_171045.log
2026-05-06T17:10:46Z [211ms] [Warning] [omni.usd_config.extension] Enable omni.materialx.libs extension to use MaterialX
2026-05-06T17:10:46Z [464ms] [Warning] [carb] Acquiring non optional plugin interface which is not listed as dependency: [omni::physx::IPhysxBenchmarks v1.0] (plugin: <default plugin>), by client: omni.physics.physx.plugin. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client.
2026-05-06T17:10:46Z [586ms] [Warning] [omni.isaac.dynamic_control] omni.isaac.dynamic_control is deprecated as of Isaac Sim 4.5. No action is needed from end-users.

|---------------------------------------------------------------------------------------------|
| Driver Version: 580.126.09    | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA GB10                      | Yes: 0 |     | 91877   MB | 10de      | 0          |
|     |                                  |        |     |            | 2e12      | 1879c72a.. |
|     |                                  |        |     |            | 1         |            |
|=============================================================================================|
| OS: 24.04.4 LTS (Noble Numbat) ubuntu, Version: 24.04.4, Kernel: 6.17.0-1008-nvidia
| XServer Vendor: The X.Org Foundation, XServer Version: 12101011 (1.21.1.11)
| Processor: Cortex-A725
| Cores: 20 | Logical Cores: 20
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 122502 | Free Memory: 116739
| Total Page/Swap (MB): 16383 | Free Page/Swap: 16383
|---------------------------------------------------------------------------------------------|
[INFO]: Parsing configuration from: isaaclab_tasks.manager_based.locomotion.velocity.config.anymal_c.rough_env_cfg:AnymalCRoughEnvCfg
[INFO]: Parsing configuration from: isaaclab_tasks.manager_based.locomotion.velocity.config.anymal_c.agents.rsl_rl_ppo_cfg:AnymalCRoughPPORunnerCfg
[WARNING]: The `policy` configuration is deprecated for rsl-rl >= 4.0.0. Please use, e.g., `actor` and `critic` model configurations instead.
[WARNING]: The `policy` configuration is used to infer the `actor` model configuration.
[WARNING]: The `policy` configuration is used to infer the `critic` model configuration.
[WARNING]: The `distribution_cfg` configuration is now used to specify the output distribution for stochastic policies. Consider updating the configuration to use `distribution_cfg` instead of `stochastic`, `init_noise_std`, `noise_std_type`, and `state_dependent_std` parameters.
[INFO] Logging experiment in directory: /home1/zhanghui/IsaacLab/logs/rsl_rl/anymal_c_rough
Exact experiment name requested from command line: 2026-05-06_17-10-51

======================================================================================
[INFO][IsaacLab]: Logging to file: /tmp/isaaclab/logs/isaaclab_2026-05-06_17-10-51.log
======================================================================================

17:10:51 [simulation_context.py] WARNING: The `enable_external_forces_every_iteration` parameter in the PhysxCfg is set to False. If you are experiencing noisy velocities, consider enabling this flag. You may need to slightly increase the number of velocity iterations (setting it to 1 or 2 rather than 0), together with this flag, to improve the accuracy of velocity updates.
[INFO]: Base environment:
	Environment device    : cuda:0
	Environment seed      : 42
	Physics step-size     : 0.005
	Rendering step-size   : 0.02
	Environment step-size : 0.02
[INFO] Generating terrains based on curriculum took : 0.868488 seconds
[INFO]: Time taken for scene creation : 5.146176 seconds
[INFO]: Scene manager:  <class InteractiveScene>
	Number of environments: 4096
	Environment spacing   : 2.5
	Source prim name      : /World/envs/env_0
	Global prim paths     : ['/World/ground']
	Replicate physics     : True
[INFO]: Starting the simulation. This may take a few seconds. Please wait...
[INFO]: Time taken for simulation start : 5.165221 seconds
[INFO] Command Manager:  <CommandManager> contains 1 active terms.
+------------------------------------------------+
|              Active Command Terms              |
+-------+---------------+------------------------+
| Index | Name          |          Type          |
+-------+---------------+------------------------+
|   0   | base_velocity | UniformVelocityCommand |
+-------+---------------+------------------------+

[INFO] Event Manager:  <EventManager> contains 3 active terms.
+--------------------------------------+
| Active Event Terms in Mode: 'startup' |
+----------+---------------------------+
|  Index   | Name                      |
+----------+---------------------------+
|    0     | physics_material          |
|    1     | add_base_mass             |
|    2     | base_com                  |
+----------+---------------------------+
+---------------------------------------+
|  Active Event Terms in Mode: 'reset'  |
+--------+------------------------------+
| Index  | Name                         |
+--------+------------------------------+
|   0    | base_external_force_torque   |
|   1    | reset_base                   |
|   2    | reset_robot_joints           |
+--------+------------------------------+
+----------------------------------------------+
|    Active Event Terms in Mode: 'interval'    |
+-------+------------+-------------------------+
| Index | Name       | Interval time range (s) |
+-------+------------+-------------------------+
|   0   | push_robot |       (10.0, 15.0)      |
+-------+------------+-------------------------+

[INFO] Recorder Manager:  <RecorderManager> contains 0 active terms.
+---------------------+
| Active Recorder Terms |
+-----------+---------+
|   Index   | Name    |
+-----------+---------+
+-----------+---------+

[INFO] Action Manager:  <ActionManager> contains 1 active terms.
+------------------------------------+
|  Active Action Terms (shape: 12)   |
+--------+-------------+-------------+
| Index  | Name        |   Dimension |
+--------+-------------+-------------+
|   0    | joint_pos   |          12 |
+--------+-------------+-------------+

Error executing job with overrides: []
Traceback (most recent call last):
  File "/home1/zhanghui/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/utils/hydra.py", line 100, in hydra_main
    func(env_cfg, agent_cfg, *args, **kwargs)
  File "/home1/zhanghui/IsaacLab/scripts/reinforcement_learning/rsl_rl/train.py", line 173, in main
    env = gym.make(args_cli.task, cfg=env_cfg, render_mode="rgb_array" if args_cli.video else None)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zhanghui/anaconda3/envs/issaacsim311/lib/python3.11/site-packages/gymnasium/envs/registration.py", line 734, in make
    env = env_creator(**env_spec_kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home1/zhanghui/IsaacLab/source/isaaclab/isaaclab/envs/manager_based_rl_env.py", line 80, in __init__
    super().__init__(cfg=cfg)
  File "/home1/zhanghui/IsaacLab/source/isaaclab/isaaclab/envs/manager_based_env.py", line 180, in __init__
    self.load_managers()
  File "/home1/zhanghui/IsaacLab/source/isaaclab/isaaclab/envs/manager_based_rl_env.py", line 117, in load_managers
    super().load_managers()
  File "/home1/zhanghui/IsaacLab/source/isaaclab/isaaclab/envs/manager_based_env.py", line 320, in load_managers
    self.observation_manager = ObservationManager(self.cfg.observations, self)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home1/zhanghui/IsaacLab/source/isaaclab/isaaclab/managers/observation_manager.py", line 83, in __init__
    super().__init__(cfg, env)
  File "/home1/zhanghui/IsaacLab/source/isaaclab/isaaclab/managers/manager_base.py", line 177, in __init__
    self._prepare_terms()
  File "/home1/zhanghui/IsaacLab/source/isaaclab/isaaclab/managers/observation_manager.py", line 558, in _prepare_terms
    obs_dims = tuple(term_cfg.func(self._env, **term_cfg.params).shape)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home1/zhanghui/IsaacLab/source/isaaclab/isaaclab/envs/utils/io_descriptors.py", line 216, in wrapper
    out = func(env, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home1/zhanghui/IsaacLab/source/isaaclab/isaaclab/envs/mdp/observations.py", line 68, in base_ang_vel
    return asset.data.root_ang_vel_b
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home1/zhanghui/IsaacLab/source/isaaclab/isaaclab/assets/articulation/articulation_data.py", line 1048, in root_ang_vel_b
    return self.root_com_ang_vel_b
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home1/zhanghui/IsaacLab/source/isaaclab/isaaclab/assets/articulation/articulation_data.py", line 837, in root_com_ang_vel_b
    return math_utils.quat_apply_inverse(self.root_link_quat_w, self.root_com_ang_vel_w)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch)

nvrtc compilation failed: 

#define NAN __int_as_float(0x7fffffff)
#define POS_INFINITY __int_as_float(0x7f800000)
#define NEG_INFINITY __int_as_float(0xff800000)


template<typename T>
__device__ T maximum(T a, T b) {
  return isnan(a) ? a : (a > b ? a : b);
}

template<typename T>
__device__ T minimum(T a, T b) {
  return isnan(a) ? a : (a < b ? a : b);
}

extern "C" __global__
void fused_mul_sub_add(float* tv_, float* tvec_7, float* tv__, float* tt_1, float* aten_add) {
{
  float v = __ldg(tvec_7 + ((long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)) % 3ll + 2ll * ((((long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)) / 3ll) * 3ll));
  float v_1 = __ldg(tv__ + 7ll * (((long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)) / 3ll));
  float v_2 = __ldg(tt_1 + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x));
  float v_3 = __ldg(tv_ + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x));
  aten_add[(long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)] = (v - v_1 * v_2) + v_3;
}
}


Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
2026-05-06T17:11:02Z [16,749ms] [Warning] [omni.physx.plugin] USD stage detach not called, holding a loose ptr to a stage!
2026-05-06T17:11:03Z [17,545ms] [Warning] [carb] Recursive unloadAllPlugins() detected!
(issaacsim311) zhanghui@gx10-8e22:/home1/zhanghui/IsaacLab$ 

[zhanghui-china]

when i run the script ,it shows this windows, and crash.

[zhanghui-china]

Enviromenrt:

DGX Spark
NVIDIA DGX Spark Version 7.4.0 (GNU/Linux 6.17.0-1008-nvidia aarch64)
Source built IsaacSim
Source built IsaacLab
conda enviroment ,python=3.11.15

all step refer to : url

[huan-zz3]

建议换用x86架构的机子。GB10的CPU是ARM架构的，pytorch和isaacsim可能有兼容性问题。

[RandomOakForest]

Thank you for posting this. Are you able to verify your Isaac Sim installation? is your Isaac Lab installation working? Do the following, as in the docs, run for you?

# Option 1: Using the isaaclab.sh executable
# note: this works for both the bundled python and the virtual environment
./isaaclab.sh -p scripts/tutorials/00_sim/create_empty.py

# Option 2: Using python in your virtual environment
python scripts/tutorials/00_sim/create_empty.py

Please post which versions of Isaac Sim and Isaac Lab are you using. Thanks.

[zhanghui-china]

my IsaacSim install step:

1. install g++ 11 and gcc 11

sudo apt-get install gcc-11 g++-11
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 200
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-11 200
gcc --version
g++ --version

2. source code compile iasscsim

git clone https://github.com/isaac-sim/IsaacSim.git isaacsim
cd isaacsim
git lfs install
git lfs pull
./build.sh

3.start iasscsim

cd _build/linux-aarch64/release
./isaac-sim.sh

[zhanghui-china]

my IsaacLab install step:

1.create conda enviroment

conda create -n isaacsim311 python=3.11 -y
conda activate isaacsim311
pip install --upgrade pip
git clone https://github.com/isaac-sim/IsaacLab.git
cd IsaacLab

ln -s /home1/zhanghui/isaacsim/_build/linux-aarch64/release _isaac_sim

2.source code compile isaacLab

cd /home1/zhanghui/IsaacLab
./isaaclab.sh --install

3. verify isaaclab install

source _isaac_sim/setup_conda_env.sh 
./isaaclab.sh -p scripts/environments/list_envs.py

export LD_PRELOAD="$LD_PRELOAD:/lib/aarch64-linux-gnu/libgomp.so.1"
./isaaclab.sh -p scripts/tutorials/00_sim/create_empty.py

[zhanghui-china]

and when i run training robot:

./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Ant-v0 --headless

report those errors.

[zhanghui-china]

where can i set gpu-architecture? such as "sm121"? or "blackwell" ?

[Exokern]

nvrtc: error: invalid value for --gpu-architecture (-arch) usually means the NVRTC library that’s getting loaded does not recognize the SM value Torch is passing (common when the GPU is newer than the CUDA/NVRTC in the environment), or that TORCH_CUDA_ARCH_LIST is set to an unsupported value.

Given your log shows:

• OS: Ubuntu 24.04 aarch64
• GPU: “NVIDIA GB10”
• Driver: 580.126.09

…could you share the output of:

python -c "import torch; print('torch', torch.__version__); print('cuda', torch.version.cuda); print('device', torch.cuda.get_device_name(0)); print('cc', torch.cuda.get_device_capability(0))"
python -c "import os; print('TORCH_CUDA_ARCH_LIST=', os.environ.get('TORCH_CUDA_ARCH_LIST'))"

If the compute capability reported by get_device_capability() is newer than what the loaded libnvrtc supports, the practical fix is to upgrade to a CUDA/NVRTC toolchain (and corresponding PyTorch build) that supports that SM, or ensure Isaac Sim/Isaac Lab isn’t picking up an older libnvrtc.so from the Conda env.

If TORCH_CUDA_ARCH_LIST is set, try unsetting it for a quick check.

[RandomOakForest]

Thank you for posting this. Could you verify you have PyTorch with CUDA 13.0 installed? This may resolve the issue.

[zhanghui-china]

python -c "import torch; print('torch', torch.version); print('cuda', torch.version.cuda); print('device', torch.cuda.get_device_name(0)); print('cc', torch.cuda.get_device_capability(0))"

(issaacsim311) zhanghui@gx10-8e22:~$ python -c "import torch; print('torch', torch.__version__); print('cuda', torch.version.cuda); print('device', torch.cuda.get_device_name(0)); print('cc', torch.cuda.get_device_capability(0))"
torch 2.9.0+cu130
cuda 13.0
/home/zhanghui/anaconda3/envs/issaacsim311/lib/python3.11/site-packages/torch/cuda/__init__.py:283: UserWarning:
    Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.
    Minimum and Maximum cuda capability supported by this version of PyTorch is
    (8.0) - (12.0)

  warnings.warn(
device NVIDIA GB10
cc (12, 1)
(issaacsim311) zhanghui@gx10-8e22:~$

(issaacsim311) zhanghui@gx10-8e22:~$ python -c "import os; print('TORCH_CUDA_ARCH_LIST=', os.environ.get('TORCH_CUDA_ARCH_LIST'))"
TORCH_CUDA_ARCH_LIST= None
(issaacsim311) zhanghui@gx10-8e22:~$

should i upgrade pytorch ?

BTW, when i run other LLM script, it also reports this warning:

  Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.
    Minimum and Maximum cuda capability supported by this version of PyTorch is
    (8.0) - (12.0)

but run scripts successfully.

[zhanghui-china]

Thank you for posting this. Could you verify you have PyTorch with CUDA 13.0 installed? This may resolve the issue.

torch 2.9.0+cu130
cuda 13.0

(issaacsim311) zhanghui@gx10-8e22:~$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Aug_20_01:57:39_PM_PDT_2025
Cuda compilation tools, release 13.0, V13.0.88
Build cuda_13.0.r13.0/compiler.36424714_0
(issaacsim311) zhanghui@gx10-8e22:~$

[RandomOakForest]

Thank you for following up. The current build you have should support this. The nvidia-smi error under cuda 13.0 is not something we would expect. Can you try other workflows with this new torch/cuda? can you verify your Isaac Sim/Lab installation works as before? Can you try a clean reinstall to be sure the setup picks up cu13 well?