RuntimeError: dr.while_loop() crash on NVIDIA H100 with Sionna RT (cuda_ad_rgb) backend

[natanzi]

Hi everyone,

I am encountering a crash when running Sionna RT (v1.2.1) on an NVIDIA H100 NVL GPU inside a Docker container.
The issue occurs specifically when using the cuda_ad_rgb variant (CUDA + Automatic Differentiation). The simulation fails during the Channel Impulse Response (CIR) computation with a Dr.Jit loop exception.
Environment:

• GPU: NVIDIA H100 NVL (Hopper Architecture)
• Docker Image: nvcr.io/nvidia/tensorflow:24.12-tf2-py3
• Sionna Version: 1.2.1
• Mitsuba Variant: cuda_ad_rgb
• Driver Version: 580.126.09 CUDA Version: 13.0
• mitsuba: 3.7.1
• drjit: 1.2.0

Issue: The script crashes at paths.cir() with the following error:
RuntimeError: dr.while_loop(): encountered an exception ... ... in field_calculator.py, _compute_cir, dr.hint(active, ...)

Here is a snippet that reproduces the crash:

import os
import tensorflow as tf
import mitsuba as mi
os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true'
try:
    mi.set_variant('cuda_ad_rgb')
    print(" Mitsuba variant set to: cuda_ad_rgb")
except Exception as e:
    print(f" Failed to set variant: {e}")
    exit()
from sionna.rt import load_scene, PlanarArray, Transmitter, Receiver, PathSolver
import sionna.rt.scenes

scenes_path = os.path.dirname(sionna.rt.scenes.__file__)
scene_path = os.path.join(scenes_path, 'simple_reflector', 'simple_reflector.xml')
scene = load_scene(scene_path)

scene.tx_array = PlanarArray(num_rows=1, num_cols=1, pattern='iso', polarization='V')
scene.rx_array = PlanarArray(num_rows=1, num_cols=1, pattern='iso', polarization='V')
scene.add(Transmitter(name='tx', position=[0, 0, 10]))
scene.add(Receiver(name='rx', position=[10, 0, 1.5]))
solver = PathSolver()
paths = solver(scene)
print(" Computing CIR... (Crash happens here)")
try:
    a, tau = paths.cir()
    print(" CIR Computed successfully!")
except Exception as e:
    print(f"\n CRASH DETECTED:\n{e}")

Has anyone experienced this specific Dr.Jit loop issue on Hopper GPUs? Is there a recommended Dr.Jit version or workaround for H100?

Thanks.

[merlinND]

Hello @natanzi,

Please share the full error log so that we can better understand what is going wrong.

[natanzi]

Hello @merlinND ,
Thanks for your reply, here you are:
test_sionna_rt_h100_issue.py
sionna_rt_h100_issue.log

BR,
Milad

[merlinND]

It seems the issue is simply due to trying to use the cuda_ad_rgb variant instead of the cuda_ad_mono_polarized variant. I don't think it's related to H100.

Did you add the mi.set_variant() call for a specific reason? Sionna RT should manage the variants automatically (including falling back to LLVM if CUDA is not available).

[natanzi]

@merlinND Thanks, the problem is solved. Just a quick question: I’m currently using one H100, but I also have access to other GPUs like H200 and H100. Is there any way to perform ray tracing or channel genration across multiple GPUs in parallel, or via some other approach?

Best regards,
Milad

[merlinND]

Hi @natanzi,

There is no built-in support for multi-GPU, but you can start multiple processes to do work in parallel on different GPUs using CUDA_VISIBLE_DEVICES=.... We have used this method to compute very large-scale radio maps in parallel in this project: https://github.com/NVlabs/sionna-large-radio-maps

By the way, note that H100/H200 GPUs do not have RT Cores for accelerated ray tracing, unlike e.g. NVIDIA RTX PRO 6000.