[Bug]: Hang issue for offline inference when using DP #27269
Description
Your current environment
The output of python collect_env.py
Your output of `python collect_env.py` here
🐛 Describe the bug
import os
from vllm import LLM, SamplingParams
if __name__ == "__main__":
# os.environ["VLLM_ATTENTION_BACKEND"] = "FLASHINFER_MLA"
os.environ["VLLM_ATTENTION_BACKEND"] = "CUTLASS_MLA"
llm = LLM(
model="deepseek-ai/DeepSeek-V2-lite",
tensor_parallel_size=1,
data_parallel_size=2,
max_num_seqs=32,
max_model_len=8192,
enforce_eager=True,
)
sp = SamplingParams(
temperature=0.6,
top_p=1.0,
max_tokens=8,
seed=1234,
logprobs=5,
)
outs = llm.generate(["There once was a "], sp, use_tqdm=True)
print(outs)Hanging here forever
Adding requests: 100%|█████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 68.39it/s]
Processed prompts: 0%| | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.