[Bug]: vLLM takes forever to load a locally stored 7B model

[vibhas-singh]

Your current environment

PyTorch version: 2.3.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Amazon Linux 2 (x86_64)
GCC version: (GCC) 7.3.1 20180712 (Red Hat 7.3.1-17)
Clang version: Could not collect
CMake version: version 3.30.1
Libc version: glibc-2.26

Python version: 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0] (64-bit runtime)
Python platform: Linux-5.10.217-205.860.amzn2.x86_64-x86_64-with-glibc2.26
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A10G
Nvidia driver version: 535.183.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  2
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           AuthenticAMD
CPU family:          23
Model:               49
Model name:          AMD EPYC 7R32
Stepping:            0
CPU MHz:             3130.445
BogoMIPS:            5599.99
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            512K
L3 cache:            16384K
NUMA node0 CPU(s):   0-7
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save rdpid

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] onnx==1.16.1
[pip3] torch==2.3.1
[pip3] torch-model-archiver==0.7.1b20230208
[pip3] torch-workflow-archiver==0.2.13b20240516
[pip3] torchaudio==2.2.0
[pip3] torchdata==0.7.1
[pip3] torchserve==0.11.0b20240516
[pip3] torchtext==0.17.0
[pip3] torchvision==0.18.1
[pip3] transformers==4.43.3
[pip3] triton==2.3.1
[conda] aws-ofi-nccl              1.9.1           aws_efa1.26.1_0    https://aws-ml-conda.s3.us-west-2.amazonaws.com
[conda] blas                      2.116                       mkl    conda-forge
[conda] blas-devel                3.9.0            16_linux64_mkl    conda-forge
[conda] libblas                   3.9.0            16_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            16_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            16_linux64_mkl    conda-forge
[conda] liblapacke                3.9.0            16_linux64_mkl    conda-forge
[conda] mkl                       2022.1.0           h84fe81f_915    https://aws-ml-conda.s3.us-west-2.amazonaws.com
[conda] mkl-devel                 2022.1.0           ha770c72_916    conda-forge
[conda] mkl-include               2022.1.0           h84fe81f_915    conda-forge
[conda] numpy                     1.26.4          py310hb13e2d6_0    conda-forge
[conda] nvidia-nccl-cu12          2.20.5                   pypi_0    pypi
[conda] pytorch-cuda              12.1                 ha16c6d3_5    https://aws-ml-conda.s3.us-west-2.amazonaws.com
[conda] pytorch-mutex             1.0                        cuda    https://aws-ml-conda.s3.us-west-2.amazonaws.com
[conda] torch                     2.3.1                    pypi_0    pypi
[conda] torch-model-archiver      0.7.1                   py310_0    pytorch
[conda] torch-workflow-archiver   0.2.13                  py310_0    pytorch
[conda] torchaudio                2.2.0               py310_cu121    https://aws-ml-conda.s3.us-west-2.amazonaws.com
[conda] torchdata                 0.7.1                     py310    https://aws-ml-conda.s3.us-west-2.amazonaws.com
[conda] torchserve                0.11.0                  py310_0    pytorch
[conda] torchtext                 0.17.0                    py310    https://aws-ml-conda.s3.us-west-2.amazonaws.com
[conda] torchvision               0.18.1                   pypi_0    pypi
[conda] transformers              4.43.3                   pypi_0    pypi
[conda] triton                    2.3.1                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.5.3.post1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	0-7	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

🐛 Describe the bug

I have fine-tuned a llama 7B model using transformers and QLoRA.
Then I have merged the LoRA weights to the base model and saved it to the disk.
The saved weights look something like this:

Now I am trying to load these weights using the python API of vLLM using the following code:

model_path = "path/of/model"
tokenizer_path = "path/to/tokenizer"
llm = LLM(model=model_path, tokenizer=tokenizer_path, max_model_len=4096)

It is almost taking forever to load all the shards. The progress bar shows the estimated load time to be 51 mins.
Loading one shard is almost taking 9 mins.

Few additional points:

• The model weights are already stored in fp16 (evident from the total size of the model on disk) so any type conversions might not be happening.
• I am using sagemaker notebook instance so disk being slow doesn't look like the issue as well.
• When I am trying to load the model weights using transformers using the following code, the first load takes around 12 mins. All the subsequent model loads take 10-20 secs only.

model = AutoModelForCausalLM.from_pretrained(model_path, device_map="auto")

• Once I load the model using transformers and rerun the same vLLM model load code, it runs instantly instead of taking 50 mins.

[DarkLight1337]

Please go through this troubleshooting guide and see if it can resolve your issue.

[vibhas-singh]

@DarkLight1337 I tried turning on the debugging using the flags - though I am not observing anything weird.

Sharing the logs here from the console (have redacted the IP):

INFO 07-30 16:24:13 llm_engine.py:176] Initializing an LLM engine (v0.5.3) with config: model='merged_model_1', speculative_config=None, tokenizer='code-llama-7b-text-to-sql', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=merged_model_1, use_v2_block_manager=False, enable_prefix_caching=False)
WARNING 07-30 16:24:14 logger.py:146] VLLM_TRACE_FUNCTION is enabled. It will record every function executed by Python. This will slow down the code. It is suggested to be used for debugging hang or crashes only.
INFO 07-30 16:24:14 logger.py:150] Trace frame log is saved to /tmp/vllm/vllm-instance-e4e21a3b888745af8bd6ea332ced22e3/VLLM_TRACE_FUNCTION_for_process_15863_thread_140205088794432_at_2024-07-30_16:24:14.428276.log
DEBUG 07-30 16:24:17 parallel_state.py:803] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://XXX.XX.XX.XX:57753 backend=nccl
INFO 07-30 16:24:17 model_runner.py:680] Starting to load model merged_model_1...
Loading safetensors checkpoint shards:   0% Completed | 0/7 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  14% Completed | 1/7 [08:12<49:16, 492.67s/it]

The sanity check is also successful.

[DarkLight1337]

@youkaichao can you help with this?

[youkaichao]

The log shows it is loading model. So might be a disk io problem here?

[vibhas-singh]

Apologies for the naive question - but it there a way to identify the disk IO problems?
Is it possible to have a sanity check like thing for disk IO as well - like we have for Incorrect hardware/driver in the troubleshooting guide?

[Ziyi6]

Apologies for the naive question - but it there a way to identify the disk IO problems? Is it possible to have a sanity check like thing for disk IO as well - like we have for Incorrect hardware/driver in the troubleshooting guide?

Bro, did you find out the answer? I also met this slow loading problem. I use HDD but I noticed that the models were already loaded into the GPU, what the program did was always waiting

[Ziyi6]

Apologies for the naive question - but it there a way to identify the disk IO problems? Is it possible to have a sanity check like thing for disk IO as well - like we have for Incorrect hardware/driver in the troubleshooting guide?

I tried to put the models to SSD, the problem solved... The loading time reduced to 2min comparing to 60min with HDD...

[vibhas-singh]

@Ziyi6 Not really.

I am using Amazon Sagemaker Notebook instances and from what I can infer from docs it - its already using general purpose SSD for storage.
Refer: https://repost.aws/questions/QUsip6vrGlRQOhlyEjZNeEMg/ebs-volume-type-for-sagemaker-notebook-instances

I am yet to explore a way to check the actual IO speed I am getting - that is why I requested the maintainers if it possible to add a debugging script for slow IO as well.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!

[nishchaysinha]

pip freeze
aiohappyeyeballs==2.4.4
aiohttp==3.11.11
aiosignal==1.3.2
annotated-types==0.7.0
anyio==4.8.0
asttokens==3.0.0
async-timeout==5.0.1
attrs==24.3.0
certifi==2024.12.14
charset-normalizer==3.4.1
click==8.1.8
cloudpickle==3.1.1
comm==0.2.2
compressed-tensors==0.8.0
datasets==3.2.0
debugpy==1.8.12
decorator==5.1.1
dill==0.3.8
diskcache==5.6.3
distro==1.9.0
einops==0.8.0
exceptiongroup==1.2.2
executing==2.2.0
fastapi==0.115.7
filelock==3.16.1
frozenlist==1.5.0
fsspec==2024.10.0
gguf==0.10.0
h11==0.14.0
httpcore==1.0.7
httptools==0.6.4
httpx==0.28.1
huggingface-hub==0.27.1
idna==3.10
importlib_metadata==8.6.1
inquirerpy==0.3.4
interegular==0.3.3
ipykernel==6.29.5
ipython==8.31.0
jedi==0.19.2
Jinja2==3.1.4
jiter==0.8.2
jsonschema==4.23.0
jsonschema-specifications==2024.10.1
jupyter_client==8.6.3
jupyter_core==5.7.2
lark==1.2.2
llvmlite==0.44.0
lm-format-enforcer==0.10.9
MarkupSafe==2.1.5
matplotlib-inline==0.1.7
mistral_common==1.5.1
mpmath==1.3.0
msgpack==1.1.0
msgspec==0.19.0
multidict==6.1.0
multiprocess==0.70.16
nest-asyncio==1.6.0
networkx==3.4.2
numba==0.61.0
numpy==2.1.2
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-cusparselt-cu12==0.6.2
nvidia-ml-py==12.560.30
nvidia-nccl-cu12==2.21.5
nvidia-nvjitlink-cu12==12.1.105
nvidia-nvtx-cu12==12.1.105
openai==1.60.0
opencv-python-headless==4.11.0.86
outlines==0.0.46
packaging==24.2
pandas==2.2.3
parso==0.8.4
partial-json-parser==0.2.1.1.post5
pexpect==4.9.0
pfzy==0.3.4
pillow==11.0.0
platformdirs==4.3.6
prometheus-fastapi-instrumentator==7.0.2
prometheus_client==0.21.1
prompt_toolkit==3.0.50
propcache==0.2.1
protobuf==5.29.3
psutil==6.1.1
ptyprocess==0.7.0
pure_eval==0.2.3
py-cpuinfo==9.0.0
pyairports==2.1.1
pyarrow==19.0.0
pycountry==24.6.1
pydantic==2.10.5
pydantic_core==2.27.2
Pygments==2.19.1
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
pytorch-triton==3.1.0+cf34004b8a
pytz==2024.2
PyYAML==6.0.2
pyzmq==26.2.0
ray==2.41.0
referencing==0.36.1
regex==2024.11.6
requests==2.32.3
rpds-py==0.22.3
safetensors==0.5.2
sentencepiece==0.2.0
six==1.17.0
sniffio==1.3.1
stack-data==0.6.3
starlette==0.45.2
sympy==1.13.1
tiktoken==0.7.0
tokenizers==0.21.0
torch==2.6.0.dev20241112+cu121
torchaudio==2.5.0.dev20241112+cu121
torchvision==0.20.0.dev20241112+cu121
tornado==6.4.2
tqdm==4.67.1
traitlets==5.14.3
transformers==4.48.1
triton==3.1.0
typing_extensions==4.12.2
tzdata==2025.1
urllib3==2.3.0
uvicorn==0.34.0
uvloop==0.21.0
vllm==0.6.4.post1
watchfiles==1.0.4
wcwidth==0.2.13
websockets==14.2
xformers==0.0.28.post3
xxhash==3.5.0
yarl==1.18.3
zipp==3.21.0`

on Python 3.10.12

This issue has been persistent with my sagemaker instance.

one thing i've noticed is that lets say i kill my kernel midway loading the shards it tends to load faster

Loading pt checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] Loading pt checkpoint shards: 25% Completed | 1/4 [00:01<00:03, 1.29s/it] Loading pt checkpoint shards: 50% Completed | 2/4 [00:25<00:29, 14.68s/it]

take the first shard for context

I have not figured out why this issue persists, If i find something i will update this

[DarkLight1337]

I suggest following the troubleshooting guide to help debug the issue.