[Bug]: fastsafetensors in tensor parallel requires too much VRAM

[Jannik2099]

Your current environment

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.5.2
[pip3] mypy==1.18.2
[pip3] mypy-extensions==1.1.0
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.16.0
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.3.0
[pip3] nvidia-ml-py==13.580.82
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvshmem-cu12==3.3.20
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] pyzmq==27.1.0
[pip3] torch==2.9.0
[pip3] torch-c-dlpack-ext==0.1.3
[pip3] torchaudio==2.9.0
[pip3] torchvision==0.24.0
[pip3] transformers==4.57.2
[pip3] triton==3.5.0
[conda] Could not collect

🐛 Describe the bug

When using the fastsafetensors plugin, it seems that VRAM usage is significantly increased in tensor parallel mode.

I am unable to load Qwen3-VL-235B-A22B-Instruct-FP8 with the parameters from https://docs.vllm.ai/projects/recipes/en/latest/Qwen/Qwen3-VL.html

vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 \
  --tensor-parallel-size 4 \
  --limit-mm-per-prompt.video 0 \
  --async-scheduling \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 128

fails when --load-format fastsafetensors is added, on four 96GiB H100s.

Using vllm 0.11.2 and fastsafetensors 0.1.15

I've also tried reducing the context size and max-num-seqs to no avail.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[Jannik2099]

Without tensor parallelism, memory usage during model load (tested with Qwen3-VL-32B-Instruct-FP8) seems similar with and without fastsafetensors

[bbrowning]

I'm seeing substantially less system memory used during model loading when using fastsafetensors on a unified memory system (DGX Spark) that does not use tensor parallelism. So, at least on some systems, it is saving substantial memory.

[bbrowning]

Could you clarify whether you're referring to GPU memory, system memory, or both?

[Jannik2099]

Sorry, I was referring exclusively to VRAM. Memory usage during load seems so high that I can't load the model outright.

[bbrowning]

Based on the little debugging I've done, I think fastsafetensors may be trying to load all the weights on the same GPU instead of actually spreading them across your GPUs as it should be with tensor parallelism.

[Jannik2099]

I think that may be happening. I just tried a tp 2 run with Qwen3-VL-32B, and I could see the tp0 worker spike massively in cpu and memory usage during model load, while the tp1 worker remained mostly idle.

[Jannik2099]

hmm, I also get completely garbled results when prompting a tensor-parallel fastsafetensors model.

Surely this isn't just outright broken and no one noticed? The fastsafetensors repo at least documents using it in tp mode.

[bbrowning]

If you're able to build vLLM from source, I commented with one potential fix for using fastsafetensors with tensor parallelism at foundation-model-stack/fastsafetensors#36 (comment) . My hunch is it is broken using TP, at least on more recent vLLM versions.

[Jannik2099]

I'll give it a try, but I am doing this on a shared HPC system, and our GPU nodes are quite in demand - so it may take a day or two until I get a job allocation.

[bbrowning]

Testing this myself in a tensor parallel 4 scenario, I'm seeing something in fastsafetensors is using GPU memory in GPU 0 from every process. See the allocations from TP1, TP2, and TP3 workers on GPU 0 here:

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A     11133      C   VLLM::Worker_TP0                            42842MiB |
|    0   N/A  N/A     11134      C   VLLM::Worker_TP1                              424MiB |
|    0   N/A  N/A     11135      C   VLLM::Worker_TP2                              424MiB |
|    0   N/A  N/A     11136      C   VLLM::Worker_TP3                              424MiB |
|    1   N/A  N/A     11134      C   VLLM::Worker_TP1                            42842MiB |
|    2   N/A  N/A     11135      C   VLLM::Worker_TP2                            42842MiB |
|    3   N/A  N/A     11136      C   VLLM::Worker_TP3                            42842MiB |
+-----------------------------------------------------------------------------------------+

They are relatively small, but still add up to use extra memory on GPU 0. When I run with the regular safetensors load strategy (ie not fastsafetensors), I do not see this additional memory usage on GPU 0.

[Jannik2099]

I can confirm that this fixes the "garbled output" issue, so weights are actually sharded to the GPUs now.

However, I still run out of VRAM trying to load Qwen3-VL-235B-A22B-Instruct-FP8 with fastsafetensors, so the overhead seems to be non-negligible :(