[Bug]: Multi-modal inference too slow

[gohar94]

Your current environment

The output of `python collect_env.py`

INFO 04-15 02:56:47 [__init__.py:239] Automatically detected platform cuda.
Collecting environment information...
PyTorch version: 2.6.0+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.12.7 | packaged by Anaconda, Inc. | (main, Oct  4 2024, 13:27:36) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-1086-azure-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 12.6.85
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-80GB
GPU 1: NVIDIA A100-SXM4-80GB
GPU 2: NVIDIA A100-SXM4-80GB
GPU 3: NVIDIA A100-SXM4-80GB
GPU 4: NVIDIA A100-SXM4-80GB
GPU 5: NVIDIA A100-SXM4-80GB
GPU 6: NVIDIA A100-SXM4-80GB
GPU 7: NVIDIA A100-SXM4-80GB

Nvidia driver version: 550.90.07
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Byte Order:                           Little Endian
Address sizes:                        48 bits physical, 48 bits virtual
CPU(s):                               96
On-line CPU(s) list:                  0-95
Thread(s) per core:                   1
Core(s) per socket:                   48
Socket(s):                            2
NUMA node(s):                         4
Vendor ID:                            AuthenticAMD
CPU family:                           23
Model:                                49
Model name:                           AMD EPYC 7V12 64-Core Processor
Stepping:                             0
CPU MHz:                              3108.517
BogoMIPS:                             4890.88
Hypervisor vendor:                    Microsoft
Virtualization type:                  full
L1d cache:                            3 MiB
L1i cache:                            3 MiB
L2 cache:                             48 MiB
L3 cache:                             384 MiB
NUMA node0 CPU(s):                    0-23
NUMA node1 CPU(s):                    24-47
NUMA node2 CPU(s):                    48-71
NUMA node3 CPU(s):                    72-95
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Mitigation; untrained return thunk; SMT disabled
Vulnerability Spec rstack overflow:   Mitigation; safe RET, no microcode
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip rdpid

Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-cusparselt-cu12==0.6.2
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.0
[pip3] torch==2.6.0
[pip3] torchaudio==2.6.0
[pip3] torchvision==0.21.0
[pip3] transformers==4.50.0.dev0
[pip3] triton==3.2.0
[pip3] zmq==0.0.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-cublas-cu12        12.4.5.8                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.4.127                 pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
[conda] nvidia-cufft-cu12         11.2.1.3                 pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.5.147               pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.6.1.9                 pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.3.1.170               pypi_0    pypi
[conda] nvidia-cusparselt-cu12    0.6.2                    pypi_0    pypi
[conda] nvidia-ml-py              12.560.30                pypi_0    pypi
[conda] nvidia-nccl-cu12          2.21.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.4.127                 pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.4.127                 pypi_0    pypi
[conda] pyzmq                     26.2.0                   pypi_0    pypi
[conda] torch                     2.6.0                    pypi_0    pypi
[conda] torchaudio                2.6.0                    pypi_0    pypi
[conda] torchvision               0.21.0                   pypi_0    pypi
[conda] transformers              4.50.0.dev0              pypi_0    pypi
[conda] triton                    3.2.0                    pypi_0    pypi
[conda] zmq                       0.0.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.8.2
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
�[4mGPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	NIC0	NIC1	NIC2	NIC3	NIC4	NIC5	NIC6	NIC7	NIC8	CPU Affinity	NUMA Affinity	GPU NUMA ID�[0m
GPU0	 X 	NV12	NV12	NV12	NV12	NV12	NV12	NV12	NODE	NODE	SYS	SYS	SYS	SYS	SYS	SYS	SYS	24-47	1		N/A
GPU1	NV12	 X 	NV12	NV12	NV12	NV12	NV12	NV12	NODE	NODE	SYS	SYS	SYS	SYS	SYS	SYS	SYS	24-47	1		N/A
GPU2	NV12	NV12	 X 	NV12	NV12	NV12	NV12	NV12	SYS	SYS	NODE	NODE	SYS	SYS	SYS	SYS	NODE	0-23	0		N/A
GPU3	NV12	NV12	NV12	 X 	NV12	NV12	NV12	NV12	SYS	SYS	NODE	NODE	SYS	SYS	SYS	SYS	NODE	0-23	0		N/A
GPU4	NV12	NV12	NV12	NV12	 X 	NV12	NV12	NV12	SYS	SYS	SYS	SYS	NODE	NODE	SYS	SYS	SYS	72-95	3		N/A
GPU5	NV12	NV12	NV12	NV12	NV12	 X 	NV12	NV12	SYS	SYS	SYS	SYS	NODE	NODE	SYS	SYS	SYS	72-95	3		N/A
GPU6	NV12	NV12	NV12	NV12	NV12	NV12	 X 	NV12	SYS	SYS	SYS	SYS	SYS	SYS	NODE	NODE	SYS	48-71	2		N/A
GPU7	NV12	NV12	NV12	NV12	NV12	NV12	NV12	 X 	SYS	SYS	SYS	SYS	SYS	SYS	NODE	NODE	SYS	48-71	2		N/A
NIC0	NODE	NODE	SYS	SYS	SYS	SYS	SYS	SYS	 X 	NODE	SYS	SYS	SYS	SYS	SYS	SYS	SYS				
NIC1	NODE	NODE	SYS	SYS	SYS	SYS	SYS	SYS	NODE	 X 	SYS	SYS	SYS	SYS	SYS	SYS	SYS				
NIC2	SYS	SYS	NODE	NODE	SYS	SYS	SYS	SYS	SYS	SYS	 X 	NODE	SYS	SYS	SYS	SYS	NODE				
NIC3	SYS	SYS	NODE	NODE	SYS	SYS	SYS	SYS	SYS	SYS	NODE	 X 	SYS	SYS	SYS	SYS	NODE				
NIC4	SYS	SYS	SYS	SYS	NODE	NODE	SYS	SYS	SYS	SYS	SYS	SYS	 X 	NODE	SYS	SYS	SYS				
NIC5	SYS	SYS	SYS	SYS	NODE	NODE	SYS	SYS	SYS	SYS	SYS	SYS	NODE	 X 	SYS	SYS	SYS				
NIC6	SYS	SYS	SYS	SYS	SYS	SYS	NODE	NODE	SYS	SYS	SYS	SYS	SYS	SYS	 X 	NODE	SYS				
NIC7	SYS	SYS	SYS	SYS	SYS	SYS	NODE	NODE	SYS	SYS	SYS	SYS	SYS	SYS	NODE	 X 	SYS				
NIC8	SYS	SYS	NODE	NODE	SYS	SYS	SYS	SYS	SYS	SYS	NODE	NODE	SYS	SYS	SYS	SYS	 X 				

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7
  NIC8: mlx5_8

LD_LIBRARY_PATH=:/usr/local/cuda/lib64
NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

I am noticing an issue when using Gemma-3-27B (and other multimodal models like NVLM-D-72B as well). I am running with TP=4 with 4 A100 GPUs in a single node using NVLink.

I have an inference request that has around 30 images.

I notice that it takes a long time (about 1.5 mins) to complete. However during this time, mostly the GPUs are idle. The main thing that is happening is that the host DRAM usage is slowly increasing. I enabled vLLM tracing to figure out what was going on and there is a large window of time (about 30s) where there is no function call and the code is waiting on acquire_read in shm_broadcast.py. I am attaching the relevant code snippets below.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[robertgshaw2-redhat]

Any way you can share a minimal reproduction?

[DarkLight1337]

This might be an issue with the serialization of multi-modal inputs. I suggest trying out #16432

[gohar94]

This might be an issue with the serialization of multi-modal inputs. I suggest trying out #16432

It didn't help that much. I will try to get a small repro example but I have around 40 images in a request of size 896x896 using Gemma-3-27B.

[DarkLight1337]

Another thing you can try is to disable multiprocessing (VLLM_ENABLE_V1_MULTIPROCESSING=0)

[gohar94]

Another thing you can try is to disable multiprocessing (VLLM_ENABLE_V1_MULTIPROCESSING=0)

I'll try that -- should I expect any other performance changes as a result of that? Throughput should not drop, right? And I can still use multiple GPUs/tensor parallelism?

[DarkLight1337]

Actually I think VLLM_ENABLE_V1_MULTIPROCESSING only works for offline inference. If you're using online serving then the AsyncLLM and EngineCore are always in two separate processes.

[DarkLight1337]

It will drop the throughput a bit but at least it may prevent hanging requests

[gohar94]

Actually I think VLLM_ENABLE_V1_MULTIPROCESSING only works for offline inference. If you're using online serving then the AsyncLLM and EngineCore are always in two separate processes.

Yeah that won't work well for my use-case. Should we not be sharing these tensors over shared memory between processes on the same node to avoid serialization costs?

[njhill]

Like @DarkLight1337 said, #16432 should help but doesn't yet cover the extra IPC involved in the TP > 1 case.

@russellb is separately working on unifying the serialization done in that case, which means it should then also exploit the #16432 improvements. So once both of these things are done things should be much better.

At that point we can reevaluate if we want to explore some more-involved shared mem approaches.

[gohar94]

Like @DarkLight1337 said, #16432 should help but doesn't yet cover the extra IPC involved in the TP > 1 case.

@russellb is separately working on unifying the serialization done in that case, which means it should then also exploit the #16432 improvements. So once both of these things are done things should be much better.

At that point we can reevaluate if we want to explore some more-involved shared mem approaches.

Sounds good. Any rough time estimate on these changes being ready to try?

[p88h]

You may want to try and refresh #16432 again - the serialization still involved some suboptimal copying, which was quite expensive, esp. on bigger machines. Since that was happening on the client size of the IPC, it might improve your case as well (as per @njhill that case is still a bit different). But I've seen a huge uplift on a single H100 and previously it was exhibiting the same symptoms as what you described.

[gohar94]

You may want to try and refresh #16432 again - the serialization still involved some suboptimal copying, which was quite expensive, esp. on bigger machines. Since that was happening on the client size of the IPC, it might improve your case as well (as per @njhill that case is still a bit different). But I've seen a huge uplift on a single H100 and previously it was exhibiting the same symptoms as what you described.

Sure, I can try today. A single H100 meaning no tensor parallelism? I want to try with TP=4 or more so you think this will still help? Glancing over your latest commits, it seems you improved some memory copying between workers so that might help, right?

[njhill]

@gohar94 the serialization there only covers the communication between the front-end process and "core" process. But we are hoping to update the core<->TP worker communication to use that too (which only applies to TP>1 cases). The complication is that a lot of it was written originally to be very general, using pickle for arbitrary messages, so it's just taking a bit of refactoring.

[effortprogrammer]

I am also facing this issue, and I think the main problem is since there's a bug with flash attention 2 with the vision module, we need to keep on with xformers backend.