Error when using in Ray cluster

[eugr]

When running VLLM in Ray cluster across two nodes with -tp 2, getting the following error. Regular safetensor loading works:

(EngineCore_DP0 pid=2442) Process EngineCore_DP0:
(EngineCore_DP0 pid=2442) Traceback (most recent call last):
(EngineCore_DP0 pid=2442)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=2442)     self.run()
(EngineCore_DP0 pid=2442)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=2442)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 846, in run_engine_core
(EngineCore_DP0 pid=2442)     raise e
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 833, in run_engine_core
(EngineCore_DP0 pid=2442)     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=2442)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 609, in __init__
(EngineCore_DP0 pid=2442)     super().__init__(
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 102, in __init__
(EngineCore_DP0 pid=2442)     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=2442)                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=2442)     self._init_executor()
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/ray_executor.py", line 97, in _init_executor
(EngineCore_DP0 pid=2442)     self._init_workers_ray(placement_group)
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/ray_executor.py", line 371, in _init_workers_ray
(EngineCore_DP0 pid=2442)     self.collective_rpc("load_model")
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/ray_executor.py", line 493, in collective_rpc
(EngineCore_DP0 pid=2442)     return ray.get(ray_worker_outputs, timeout=timeout)
(EngineCore_DP0 pid=2442)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
(EngineCore_DP0 pid=2442)     return fn(*args, **kwargs)
(EngineCore_DP0 pid=2442)            ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
(EngineCore_DP0 pid=2442)     return func(*args, **kwargs)
(EngineCore_DP0 pid=2442)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 2972, in get
(EngineCore_DP0 pid=2442)     values, debugger_breakpoint = worker.get_objects(
(EngineCore_DP0 pid=2442)                                   ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 1031, in get_objects
(EngineCore_DP0 pid=2442)     raise value.as_instanceof_cause()
(EngineCore_DP0 pid=2442) ray.exceptions.RayTaskError(AcceleratorError): ray::RayWorkerWrapper.execute_method() (pid=877, ip=192.168.177.12, actor_id=20efcf2913ef05e274cb90e503000000, repr=<vllm.v1.executor.ray_utils.RayWorkerWrapper object at 0xe2e315e5a4e0>)
(EngineCore_DP0 pid=2442)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=2442)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/worker_base.py", line 345, in execute_method
(EngineCore_DP0 pid=2442)     raise e
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/worker_base.py", line 334, in execute_method
(EngineCore_DP0 pid=2442)     return run_method(self, method, args, kwargs)
(EngineCore_DP0 pid=2442)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/serial_utils.py", line 479, in run_method
(EngineCore_DP0 pid=2442)     return func(*args, **kwargs)
(EngineCore_DP0 pid=2442)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 262, in load_model
(EngineCore_DP0 pid=2442)     self.model_runner.load_model(eep_scale_up=eep_scale_up)
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3276, in load_model
(EngineCore_DP0 pid=2442)     self.model = model_loader.load_model(
(EngineCore_DP0 pid=2442)                  ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 55, in load_model
(EngineCore_DP0 pid=2442)     self.load_weights(model, model_config)
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/default_loader.py", line 305, in load_weights
(EngineCore_DP0 pid=2442)     loaded_weights = model.load_weights(self.get_all_weights(model_config, model))
(EngineCore_DP0 pid=2442)                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_vl.py", line 1670, in load_weights
(EngineCore_DP0 pid=2442)     return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
(EngineCore_DP0 pid=2442)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/online_quantization.py", line 173, in patched_model_load_weights
(EngineCore_DP0 pid=2442)     return original_load_weights(auto_weight_loader, weights, mapper=mapper)
(EngineCore_DP0 pid=2442)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 336, in load_weights
(EngineCore_DP0 pid=2442)     autoloaded_weights = set(self._load_module("", self.module, weights))
(EngineCore_DP0 pid=2442)                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 280, in _load_module
(EngineCore_DP0 pid=2442)     for child_prefix, child_weights in self._groupby_prefix(weights):
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 164, in _groupby_prefix
(EngineCore_DP0 pid=2442)     for prefix, group in itertools.groupby(weights_by_parts, key=lambda x: x[0][0]):
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 159, in <genexpr>
(EngineCore_DP0 pid=2442)     weights_by_parts = (
(EngineCore_DP0 pid=2442)                        ^
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 332, in <genexpr>
(EngineCore_DP0 pid=2442)     weights = (
(EngineCore_DP0 pid=2442)               ^
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 90, in <genexpr>
(EngineCore_DP0 pid=2442)     return (
(EngineCore_DP0 pid=2442)            ^
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/default_loader.py", line 277, in get_all_weights
(EngineCore_DP0 pid=2442)     yield from self._get_weights_iterator(primary_weights)
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/default_loader.py", line 263, in <genexpr>
(EngineCore_DP0 pid=2442)     return ((source.prefix + name, tensor) for (name, tensor) in weights_iterator)
(EngineCore_DP0 pid=2442)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/weight_utils.py", line 748, in fastsafetensors_weights_iterator
(EngineCore_DP0 pid=2442)     fb = loader.copy_files_to_device()
(EngineCore_DP0 pid=2442)          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/fastsafetensors/loader.py", line 135, in copy_files_to_device
(EngineCore_DP0 pid=2442)     self.framework.set_device(self.device)
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/fastsafetensors/frameworks/_torch.py", line 148, in set_device
(EngineCore_DP0 pid=2442)     torch.cuda.set_device(device.as_str())
(EngineCore_DP0 pid=2442)   File "/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py", line 567, in set_device
(EngineCore_DP0 pid=2442)     torch._C._cuda_setDevice(device)
(EngineCore_DP0 pid=2442) torch.AcceleratorError: CUDA error: invalid device ordinal
(EngineCore_DP0 pid=2442) GPU device may be out of range, do you have enough GPUs?
(EngineCore_DP0 pid=2442) CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=2442) For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=2442) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=2442) INFO 11-25 00:52:57 [ray_executor.py:121] Shutting down Ray distributed executor. If you see error log from logging.cc regarding SIGTERM received, please ignore because this is the expected termination process in Ray.

[bbrowning]

Looking at how this is integrated into vLLM today, I'm not entirely sure it's setup to work within distributed vLLM properly. Something like this in the vLLM code might get things working on Ray, but I'm not sure it's entirely correct or will handle all cases of vLLM distributed.

diff --git a/vllm/model_executor/model_loader/weight_utils.py b/vllm/model_executor/model_loader/weight_utils.py
index 0809bdfa9..a7878f44f 100644
--- a/vllm/model_executor/model_loader/weight_utils.py
+++ b/vllm/model_executor/model_loader/weight_utils.py
@@ -28,6 +28,7 @@ from vllm import envs
 from vllm.config import ModelConfig
 from vllm.config.load import LoadConfig
 from vllm.distributed import get_tensor_model_parallel_rank
+from vllm.distributed.parallel_state import get_world_group
 from vllm.logger import init_logger
 from vllm.model_executor.layers.quantization import (
     QuantizationConfig,
@@ -770,11 +771,13 @@ def fastsafetensors_weights_iterator(
     """Iterate over the weights in the model safetensor files
     using fastsafetensor library."""
     if torch.distributed.is_initialized():
-        pg = torch.distributed.group.WORLD
+        world = get_world_group()
+        pg = world.device_group
+        device = world.device
     else:
         pg = SingleGroup()
+        device = torch.device(f"cuda:{pg.rank()}")
 
-    device = torch.device(f"cuda:{pg.rank()}")
     weight_files_sub_lists = [
         hf_weights_files[i : i + pg.size()]
         for i in range(0, len(hf_weights_files), pg.size())

The way weight_files_sub_lists is calculated makes me wonder if that will always load exactly the weights needed under various combinations of vLLM tensor parallelization. But, it's worth a shot for this simple case if you can easily try it.

[bbrowning]

Note that if this fixes things, this would end up being a change we need to make in vLLM itself vs in the fastsafetensors project.

[eugr]

@bbrowning - tested on gpt-oss-120b in cluster configuration - model loading improved substantially, everything seems to be working so far, and I'm getting the same benchmarks as with regular loading mechanism. Will incorporate it into my build, and continue testing on other models.

[takeshi-yoshimura]

Thank you for reporting the issue, but honestly speaking, this week and early next week I cannot find my time to work on this. So, it would be nice if you could investigate where the root cause is, fastsafetensors, vllm, ray, or whatever.

[eugr]

@takeshi-yoshimura - no problem, looks like @bbrowning pinpointed the issue, and it seems like it could be fixed on VLLM side. We will do more extensive testing and let you know! I'll keep the issue open for now, if you don't mind.

[eugr]

@bbrowning - so far so good. Tested Qwen3-VL-235B AWQ and NVFP4, MiniMax M2 AWQ, gpt-oss-120b in cluster configuration on Dual Sparks - works well.

The only issue I've had so far is when I tried to load GLM 4.6 NVFP4 - both of my Sparks ran out of memory and crashed, so I had to power cycle them. Interestingly, that didn't happen when I used normal safetensors loading - it loaded the model, but then complained about insufficient memory for KV cache and quit, but at least it didn't crash. In both cases I had max memory utilization set to 0.8, so I'd expect an error, but not crash.

[bbrowning]

@eugr I don't believe fastsafetensors knows about or respects vLLM's gpu-memory-utilization parameter. Also, be aware of #38 as it looks like on a DGX Spark using fastsafetensors often corrupts weights leading to nonsensical model outputs. It doesn't happen for every model, and likely has something to do with the specific size and memory boundaries of each model's tensors. That can be worked around for now by hardcoding nogds=True in the vLLM integration code (similar place as my patch above). I'll submit a PR to vLLM today to get these changes made by default.

Also, specific to memory usage, see #30. There's not a fastsafetensors release with that fix to disable the cache yet, but that will be a crucial fix to use fastsafetensors with larger models on most systems.

[eugr]

@bbrowning - I replied in that thread, but I don't see this behavior on my system, and GDS stays disabled, at least with fastsafetensors from PyPi.

But then, the last release was in July, and I haven't tried the main branch yet. The cache improvement is substantial and explains why I'm getting crashes when loading large models.

[takeshi-yoshimura]

The only issue I've had so far is when I tried to load GLM 4.6 NVFP4 - both of my Sparks ran out of memory and crashed, so I had to power cycle them. Interestingly, that didn't happen when I used normal safetensors loading - it loaded the model, but then complained about insufficient memory for KV cache and quit, but at least it didn't crash. In both cases I had max memory utilization set to 0.8, so I'd expect an error, but not crash.

I don't believe fastsafetensors knows about or respects vLLM's gpu-memory-utilization parameter.

Yes, fastsfafetensors allocates memory from torch CUDA caching allocator, but it does not work along with vllm. Let me think how to deal with this...

[takeshi-yoshimura]

@eugr
I could not reproduce your failure (so I updated the version to 0.2.0 without this fix). In my env with 2 or 4 H100 nodes, it said CUDA OoM and crashed. If possible, could you show a script to reproduce?

Also, it could depend on the timing of OoM. How was your vllm log like?