Issues running IsaacLab locally: Could not initialize NVML

[kpnivya]

Hi, I installed IsaacLab a couple of months ago following the instructions in the quick installation guide. I also had IsaacSim with ROS2 Humble bridge enabled before installing IsaacLab. I remember running the create_empty.py tutorial with no issues. However, I haven't really gotten around to playing with IsaacLab until now, so I tried running IsaacLab examples using python scripts/reinforcement_learning/skrl/train.py --task=Isaac-Ant-v0. This showed an empty IsaacSim gui but with no ground. There was also this error in the terminal:

2025-10-25 23:45:10 [14,830ms] [Error] [omni.ext._impl._internal] Failed to import python module isaaclab_tasks from /home/IsaacLab/source/isaaclab_tasks. Error: operator torchvision::nms does not exist. Traceback:
Traceback (most recent call last):
  File "/home/miniconda3/envs/env_isaaclab/lib/python3.10/site-packages/omni/kernel/py/omni/ext/_impl/_internal.py", line 187, in _custom_importer
    return _import_public(ext_module.path, ext_module.name, reload_enabled)
  File "/home/miniconda3/envs/env_isaaclab/lib/python3.10/site-packages/omni/kernel/py/omni/ext/_impl/_internal.py", line 94, in _import_public
    module = import_module(module_name)
  File "/home/miniconda3/envs/env_isaaclab/lib/python3.10/site-packages/omni/kernel/py/omni/ext/_impl/custom_importer.py", line 76, in import_module
    return importlib.import_module(name)
  File "/home/miniconda3/envs/env_isaaclab/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/__init__.py", line 31, in <module>
    import_packages(__name__, _BLACKLIST_PKGS)
  File "/home/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/utils/importer.py", line 40, in import_packages
    for _ in _walk_packages(package.__path__, package.__name__ + ".", blacklist_pkgs=blacklist_pkgs):
  File "/home/saacLab/source/isaaclab_tasks/isaaclab_tasks/utils/importer.py", line 87, in _walk_packages
    yield from _walk_packages(path, info.name + ".", onerror, blacklist_pkgs)
  File "/home/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/utils/importer.py", line 87, in _walk_packages
    yield from _walk_packages(path, info.name + ".", onerror, blacklist_pkgs)
  File "/home/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/utils/importer.py", line 87, in _walk_packages
    yield from _walk_packages(path, info.name + ".", onerror, blacklist_pkgs)
  [Previous line repeated 1 more time]
  File "/home/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/utils/importer.py", line 75, in _walk_packages
    __import__(info.name)
  File "/home/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack/config/franka/__init__.py", line 8, in <module>
    from . import (
  File "/home/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack/config/franka/stack_ik_rel_blueprint_env_cfg.py", line 8, in <module>
    from torchvision.utils import save_image
  File "/home/miniconda3/envs/env_isaaclab/lib/python3.10/site-packages/torchvision/__init__.py", line 10, in <module>
    from torchvision import _meta_registrations, datasets, io, models, ops, transforms, utils  # usort:skip
  File "/home/miniconda3/envs/env_isaaclab/lib/python3.10/site-packages/torchvision/_meta_registrations.py", line 164, in <module>
    def meta_nms(dets, scores, iou_threshold):
  File "/home/.local/lib/python3.10/site-packages/torch/library.py", line 1063, in register
    use_lib._register_fake(
  File "/home/.local/lib/python3.10/site-packages/torch/library.py", line 211, in _register_fake
    handle = entry.fake_impl.register(
  File "/home/.local/lib/python3.10/site-packages/torch/_library/fake_impl.py", line 50, in register
    if torch._C._dispatch_has_kernel_for_dispatch_key(self.qualname, "Meta"):
RuntimeError: operator torchvision::nms does not exist

2025-10-25 23:45:10 [14,830ms] [Error] [carb.scripting-python.plugin] Exception: Extension python module: 'isaaclab_tasks' in '/home/IsaacLab/source/isaaclab_tasks' failed to load.

At:
  /home/miniconda3/envs/env_isaaclab/lib/python3.10/site-packages/omni/kernel/py/omni/ext/_impl/_internal.py(213): startup
  /home/miniconda3/envs/env_isaaclab/lib/python3.10/site-packages/omni/kernel/py/omni/ext/_impl/_internal.py(328): startup_extension
  PythonExtension.cpp::startup()(2): <module>
  /home/miniconda3/envs/env_isaaclab/lib/python3.10/site-packages/isaacsim/exts/isaacsim.simulation_app/isaacsim/simulation_app/simulation_app.py(427): _start_app
  /home/miniconda3/envs/env_isaaclab/lib/python3.10/site-packages/isaacsim/exts/isaacsim.simulation_app/isaacsim/simulation_app/simulation_app.py(212): __init__
  /home/IsaacLab/source/isaaclab/isaaclab/app/app_launcher.py(780): _create_app
  /home/IsaacLab/source/isaaclab/isaaclab/app/app_launcher.py(131): __init__
  /home/IsaacLab/scripts/reinforcement_learning/skrl/train.py(60): <module>

2025-10-25 23:45:10 [14,830ms] [Error] [omni.ext.plugin] [ext: isaaclab_tasks-0.10.31] Failed to startup python extension.

Following the discussion in this forum topic, I uninstalled ROS2 Humble:
sudo apt remove ~nros-humble-* && sudo apt autoremove
sudo apt remove ros2-apt-source
sudo apt update
sudo apt autoremove
sudo apt upgrade
I confirm that ros2 is gone using ls /opt/

Then tried running the same Ant-v0 script again but got:

2025-10-26 00:02:22 [571ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2025-10-26 00:02:22 [571ms] [Error] [carb.graphics-vulkan.plugin] vkEnumeratePhysicalDevices failed. No physical device is found.
2025-10-26 00:02:22 [571ms] [Error] [carb.graphics-vulkan.plugin] No physical device is found.
2025-10-26 00:02:22 [573ms] [Error] [gpu.foundation.plugin] carb::graphics::createInstance failed.
2025-10-26 00:02:22 [657ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2025-10-26 00:02:22 [657ms] [Error] [carb.graphics-vulkan.plugin] vkEnumeratePhysicalDevices failed. No physical device is found.
2025-10-26 00:02:22 [657ms] [Error] [carb.graphics-vulkan.plugin] No physical device is found.
2025-10-26 00:02:22 [659ms] [Error] [gpu.foundation.plugin] carb::graphics::createInstance failed.
2025-10-26 00:02:22 [659ms] [Error] [omni.gpu_foundation_factory.plugin] Failed to create any GPU devices, including an attempt with compatibility mode.
[7.892s] [ext: omni.kit.pipapi-0.0.0] startup
[7.893s] [ext: omni.materialx.libs-1.0.6] startup
[7.899s] [ext: omni.kit.pip_archive-0.0.0] startup
[7.900s] [ext: omni.usd.config-1.0.5] startup
[7.904s] [ext: omni.gpucompute.plugins-0.0.0] startup
[7.905s] [ext: omni.usd.libs-1.0.1] startup
[7.958s] [ext: omni.mdl-55.0.1] startup
[7.971s] [ext: omni.iray.libs-0.0.0] startup
[7.975s] [ext: omni.mdl.neuraylib-0.2.10] startup
[7.977s] [ext: omni.kit.usd.mdl-1.0.7] startup
[8.046s] [ext: omni.kit.loop-isaac-1.2.1] startup
[8.047s] [ext: omni.kit.test-1.1.2] startup
[8.079s] [ext: omni.kit.telemetry-0.5.1] startup
[8.089s] [ext: omni.appwindow-1.1.9] startup
[9.170s] [ext: omni.kit.renderer.core-1.0.2] startup
2025-10-26 00:02:30 [9,155ms] [Error] [omni.kit.renderer.plugin] GPU Foundation is not initialized!
[9.173s] [ext: omni.kit.renderer.capture-0.0.0] startup
[9.175s] [ext: omni.kit.renderer.imgui-1.0.2] startup
2025-10-26 00:02:30 [9,162ms] [Error] [omni.kit.imgui_renderer.plugin] _prepareDefaultWindow: invalid device attached to the default window!


Extension modules: omni.mdl.pymdlsdk._pymdlsdk, psutil._psutil_linux, psutil._psutil_posix, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, PIL._imaging, PIL._imagingft (total: 69)
Segmentation fault (core dumped)

This time nothing ran because of the seg fault and program ended. So I deleted the repo, removed the conda env to start a fresh install following the same steps in quick installation. And now I get this error:

/IsaacLab$ python scripts/reinforcement_learning/skrl/train.py --task=Isaac-Ant-v0
[INFO][AppLauncher]: Using device: cuda:0
[INFO][AppLauncher]: Loading experience file: /home/IsaacLab/apps/isaaclab.python.kit
[Warning] [simulation_app.simulation_app] Modules: ['omni.kit_app'] were loaded before SimulationApp was started and might not be loaded correctly.
[Warning] [simulation_app.simulation_app] Please check to make sure no extra omniverse or pxr modules are imported before the call to SimulationApp(...)
[Info] [carb] Logging to file: /home/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/omni/logs/Kit/Isaac-Sim/5.0/kit_20251025_201723.log
[0.035s] [ext: omni.kit.async_engine-0.0.3] startup
[0.118s] [ext: omni.metrics.core-0.0.3] startup
[0.118s] [ext: omni.client.lib-1.1.0] startup
[0.131s] [ext: omni.blobkey-1.1.2] startup
[0.131s] [ext: omni.stats-1.0.1] startup
[0.131s] [ext: omni.datastore-0.0.0] startup
[0.136s] [ext: omni.client-1.3.0] startup
[0.140s] [ext: omni.ujitso.default-1.0.0] startup
[0.141s] [ext: omni.hsscclient-1.1.2] startup
[0.145s] [ext: omni.gpu_foundation.shadercache.vulkan-1.0.0] startup
[0.148s] [ext: omni.assets.plugins-0.0.0] startup
[0.149s] [ext: omni.gpu_foundation-0.0.0] startup
[0.158s] [ext: carb.windowing.plugins-1.0.0] startup
2025-10-26T00:17:23Z [154ms] [Warning] [carb.windowing-glfw.gamepad] Joystick with unknown remapping detected (will be ignored):  ASRock LED Controller [03000000ce260000a201000010010000]
[0.165s] [ext: omni.kit.renderer.init-0.0.0] startup
2025-10-26T00:17:23Z [177ms] [Error] [carb.cudainterop.plugin] Could not initialize NVML: return code 18 (NVML_ERROR_LIB_RM_VERSION_MISMATCH: RM detects a driver/library version mismatch.)
2025-10-26T00:17:23Z [179ms] [Error] [carb.cudainterop.plugin] Could not initialize NVML: return code 18 (NVML_ERROR_LIB_RM_VERSION_MISMATCH: RM detects a driver/library version mismatch.)
2025-10-26T00:17:24Z [253ms] [Warning] [gpu.foundation.plugin] Skipping unsupported non-NVIDIA GPU: AMD Unknown (RADV RAPHAEL_MENDOCINO)
2025-10-26T00:17:24Z [253ms] [Warning] [gpu.foundation.plugin] Skipping unsupported non-NVIDIA GPU: AMD Unknown (RADV RAPHAEL_MENDOCINO)

|---------------------------------------------------------------------------------------------|
| Driver Version: 5c02001       | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | AMD Unknown (RADV RAPHAEL_MEND.. |        |     | 21466   MB | 1002      | 0          |
|     |                                  |        |     |            | 164e      | 00000000.. |
|     |                                  |        |     |            | 71        |            |
|=============================================================================================|
| OS: 22.04.5 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.5, Kernel: 6.8.0-85-generic
| XServer Vendor: The X.Org Foundation, XServer Version: 12101004 (1.21.1.4)
| Processor: AMD Ryzen 7 7800X3D 8-Core Processor
| Cores: 8 | Logical Cores: 16
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 63375 | Free Memory: 56993
| Total Page/Swap (MB): 2047 | Free Page/Swap: 2047
|---------------------------------------------------------------------------------------------|
2025-10-26T00:17:24Z [255ms] [Error] [gpu.foundation.plugin] No device could be created. Some known system issues:
- The driver is not installed properly and requires a clean re-install.
- Your GPUs do not support RayTracing: DXR or Vulkan ray_tracing, or hardware is excluded due to performance.
- The driver cannot enumerate any GPU: driver, display, TCC mode or a docker issue. For Vulkan, test it with Vulkaninfo tool from Vulkan SDK, instead of nvidia-smi.
- For Ubuntu, it requires server-xorg-core 1.20.7+ and a display to work without --no-window.
- For Linux dockers, the setup is not complete. Install the latest driver, xServer and NVIDIA container runtime.

2025-10-26T00:17:24Z [263ms] [Error] [carb.cudainterop.plugin] Could not initialize NVML: return code 18 (NVML_ERROR_LIB_RM_VERSION_MISMATCH: RM detects a driver/library version mismatch.)
2025-10-26T00:17:24Z [265ms] [Error] [carb.cudainterop.plugin] Could not initialize NVML: return code 18 (NVML_ERROR_LIB_RM_VERSION_MISMATCH: RM detects a driver/library version mismatch.)

|---------------------------------------------------------------------------------------------|
| Driver Version: 5c02001       | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | AMD Unknown (RADV RAPHAEL_MEND.. | Yes: 0 |     | 21466   MB | 1002      | 0          |
|     |                                  |        |     |            | 164e      | 00000000.. |
|     |                                  |        |     |            | 71        |            |
|=============================================================================================|
| OS: 22.04.5 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.5, Kernel: 6.8.0-85-generic
| XServer Vendor: The X.Org Foundation, XServer Version: 12101004 (1.21.1.4)
| Processor: AMD Ryzen 7 7800X3D 8-Core Processor
| Cores: 8 | Logical Cores: 16
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 63375 | Free Memory: 56981
| Total Page/Swap (MB): 2047 | Free Page/Swap: 2047
|---------------------------------------------------------------------------------------------|
2025-10-26T00:17:24Z [438ms] [Warning] [gpu.foundation.plugin] CPU performance profile is set to powersave. This profile sets the CPU to the lowest frequency reducing performance.
2025-10-26T00:17:24Z [438ms] [Warning] [gpu.foundation.plugin] IOMMU is enabled.
2025-10-26T00:17:24Z [438ms] [Warning] [omni.gpu_foundation_factory.plugin] RT-capable GPU not found, switching to compatibility mode
[0.454s] [ext: omni.materialx.libs-1.0.7] startup
[0.460s] [ext: omni.usd.config-1.0.6] startup
[0.463s] [ext: omni.gpucompute.plugins-0.0.0] startup
[0.463s] [ext: omni.usd.libs-1.0.1] startup
[0.529s] [ext: omni.mdl-56.0.3] startup
[0.595s] [ext: omni.iray.libs-0.0.0] startup
[0.600s] [ext: omni.mdl.neuraylib-0.2.12] startup
[0.603s] [ext: omni.kit.usd.mdl-1.1.4] startup
[0.673s] [ext: omni.kit.telemetry-0.5.2] startup
[0.685s] [ext: omni.kit.pipapi-0.0.0] startup
[0.685s] [ext: omni.kit.loop-isaac-1.3.7] startup
[0.686s] [ext: omni.kit.test-2.0.1] startup
[0.725s] [ext: omni.appwindow-1.1.10] startup
[1.231s] [ext: omni.kit.renderer.core-1.1.0] startup
X Error of failed request:  BadMatch (invalid parameter attributes)
  Major opcode of failed request:  149 ()
  Minor opcode of failed request:  4
  Serial number of failed request:  360
  Current serial number in output stream:  378

How do I get IsaacLab to run locally? Please let me know if you need any additional information. Thanks!