[Release]: Release checklist for `v0.9.1rc3`

[shen-shanshan]

Release Checklist

Release Version: v0.9.1rc3

Release Branch: v0.9.1-dev

Release Date: 2025/08/20

Release Manager: @shen-shanshan

Prepare Release Note

• Create a new issue for release feedback [v0.9.1rc3] FAQ / Feedback | 问题/反馈 #2410
• Write the release note PR [0.9.1][Doc] Add release note for v0.9.1rc3 #2431
  • Update the feedback issue link in docs/source/faqs.md
  • Add release note to docs/source/user_guide/release_notes.md
  • Update version info in docs/source/community/versioning_policy.md
  • Update contributor info in docs/source/community/contributors.md
  • Update package version in docs/conf.py

PR need Merge

• [Bugfix] Fix grammar_bitmask IndexError caused by outdated apply_grammar_bitmask method #2314
• [Bugfix] Fix the bug that qwen3 moe doesn't work with aclgraph #2478

Functional Test

• DeepSeek W8A8 MTP with V1 Scheduler (A3 DP4 TP4) @Potabk
• In DeepSeek-R1 W8A8 PD disagregated Decode instance, using pure DP, with lmhead_tensor_parallel_size=8 @zhangxinyuehfad
• DeepSeek with V1 scheduler (with chunked prefill enabled) @MengqingCao

--additional_config={"lmhead_tensor_parallel_size": 8}

Doc Test

• Tutorial is updated.
• User Guide is updated.
• Developer Guide is updated.

Prepare Artifacts

• Docker image is ready.
• Wheel package is ready.

Release Step

• Release note PR is merged.
• Post the release on GitHub release page.
• Generate official doc page on https://app.readthedocs.org/dashboard/
• Wait for the wheel package to be available on https://pypi.org/project/vllm-ascend
• Wait for the docker image to be available on https://quay.io/ascend/vllm-ascend
• Upload 310p wheel to Github release page
• Broadcast the release news (By message, blog , etc)
• Close this issue

[MengqingCao]

DeepSeek-V2-Lite successed with V1 scheduler (chunk prefill enabled by default)

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]

# Create a sampling params object.
sampling_params = SamplingParams(max_tokens=10, temperature=0.0)
# Create an LLM.
llm = LLM(
    model="/home/xxx/cache/modelscope/models/deepseek-ai/DeepSeek-V2-Lite-Chat",
    tensor_parallel_size=2,
    # distributed_executor_backend="mp",
    trust_remote_code=True,
    enforce_eager=True,
    # additional_config={
    #     "lmhead_tp_size": 2,
    # }
)

# Generate texts from the prompts.
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

INFO 08-21 13:21:20 [importing.py:17] Triton not installed or not compatible; certain GPU-related functions will not be available.
WARNING 08-21 13:21:20 [importing.py:29] Triton is not installed. Using dummy decorators. Install it via `pip install triton` to enable kernel compilation.
INFO 08-21 13:21:23 [__init__.py:39] Available plugins for group vllm.platform_plugins:
INFO 08-21 13:21:23 [__init__.py:41] - ascend -> vllm_ascend:register
INFO 08-21 13:21:23 [__init__.py:44] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 08-21 13:21:23 [__init__.py:235] Platform plugin ascend is activated
WARNING 08-21 13:21:27 [_custom_ops.py:22] Failed to import from vllm._C with ImportError('libnuma.so.1: cannot open shared object file: No such file or directory')
WARNING 08-21 13:21:31 [registry.py:401] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_mtp:CustomDeepSeekMTP.
WARNING 08-21 13:21:31 [registry.py:401] Model architecture Qwen2ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2:CustomQwen2ForCausalLM.
WARNING 08-21 13:21:31 [registry.py:401] Model architecture Qwen2VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_vl:AscendQwen2VLForConditionalGeneration.
WARNING 08-21 13:21:31 [registry.py:401] Model architecture Qwen2_5_VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_5_vl:AscendQwen2_5_VLForConditionalGeneration.
WARNING 08-21 13:21:31 [registry.py:401] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV2ForCausalLM.
WARNING 08-21 13:21:31 [registry.py:401] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV3ForCausalLM.
WARNING 08-21 13:21:31 [registry.py:401] Model architecture Qwen3MoeForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3_moe:CustomQwen3MoeForCausalLM.
WARNING 08-21 13:21:31 [registry.py:401] Model architecture Qwen3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3:CustomQwen3ForCausalLM.
INFO 08-21 13:21:31 [config.py:224] Replacing legacy 'type' key with 'rope_type'
INFO 08-21 13:21:51 [config.py:823] This model supports multiple tasks: {'score', 'reward', 'classify', 'generate', 'embed'}. Defaulting to 'generate'.
WARNING 08-21 13:21:51 [arg_utils.py:1647] Detected VLLM_USE_V1=1 with npu. Usage should be considered experimental. Please report any issues on Github.
INFO 08-21 13:21:51 [config.py:1946] Defaulting to use mp for distributed inference
INFO 08-21 13:21:51 [config.py:1980] Disabled the custom all-reduce kernel because it is not supported on current platform.
INFO 08-21 13:21:51 [config.py:2195] Chunked prefill is enabled with max_num_batched_tokens=8192.
INFO 08-21 13:21:51 [platform.py:181] Compilation disabled, using eager mode by default
INFO 08-21 13:21:52 [core.py:455] Waiting for init message from front-end.
INFO 08-21 13:21:52 [platform.py:181] Compilation disabled, using eager mode by default
INFO 08-21 13:21:52 [core.py:70] Initializing a V1 LLM engine (v0.9.2.dev2+g3ea35bdc8.d20250820) with config: model='/home/xxx/cache/modelscope/models/deepseek-ai/DeepSeek-V2-Lite-Chat', speculative_config=None, tokenizer='/home/xxx/cache/modelscope/models/deepseek-ai/DeepSeek-V2-Lite-Chat', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=16384, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, kv_cache_dtype=auto,  device_config=npu, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/home/xxx/cache/modelscope/models/deepseek-ai/DeepSeek-V2-Lite-Chat, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":["all"],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":0,"local_cache_dir":null}
WARNING 08-21 13:21:52 [multiproc_worker_utils.py:307] Reducing Torch parallelism from 192 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 08-21 13:21:52 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1], buffer_handle=(2, 16777216, 10, 'psm_a6636a10'), local_subscribe_addr='ipc:///tmp/6921e398-1470-4829-b44c-c6cbd1bbc3a4', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 08-21 13:21:52 [camem.py:63] Failed to import vllm_ascend_C:/home/xxx/code/vllm-ascend/vllm_ascend/vllm_ascend_C.cpython-310-aarch64-linux-gnu.so: undefined symbol: _ZN6at_npu6native9OpCommand4NameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE. Sleep mode will be disabled. 
WARNING 08-21 13:21:52 [camem.py:63] Failed to import vllm_ascend_C:/home/xxx/code/vllm-ascend/vllm_ascend/vllm_ascend_C.cpython-310-aarch64-linux-gnu.so: undefined symbol: _ZN6at_npu6native9OpCommand4NameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE. Sleep mode will be disabled. 
WARNING 08-21 13:21:53 [utils.py:2737] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes not implemented in <vllm_ascend.worker.worker_v1.NPUWorker object at 0xfffcfbb585b0>
(VllmWorker rank=0 pid=4011636) INFO 08-21 13:21:53 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_80d81c39'), local_subscribe_addr='ipc:///tmp/ef016aea-fb9e-47a1-9c77-7e8c46ba9dd1', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 08-21 13:21:53 [utils.py:2737] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes not implemented in <vllm_ascend.worker.worker_v1.NPUWorker object at 0xfffcfbb59060>
(VllmWorker rank=1 pid=4011638) INFO 08-21 13:21:53 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_02ac32ba'), local_subscribe_addr='ipc:///tmp/660c0d07-010c-4421-b0c3-9128dfa6c499', remote_subscribe_addr=None, remote_addr_ipv6=False)
(VllmWorker rank=0 pid=4011636) INFO 08-21 13:22:03 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[1], buffer_handle=(1, 4194304, 6, 'psm_e1477d3f'), local_subscribe_addr='ipc:///tmp/62704d54-7d38-487f-97cc-d60931731b3e', remote_subscribe_addr=None, remote_addr_ipv6=False)
(VllmWorker rank=0 pid=4011636) INFO 08-21 13:22:03 [parallel_state.py:1065] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(VllmWorker rank=1 pid=4011638) INFO 08-21 13:22:03 [parallel_state.py:1065] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 1, EP rank 1
(VllmWorker rank=1 pid=4011638) INFO 08-21 13:22:07 [model_runner_v1.py:1896] Starting to load model /home/xxx/cache/modelscope/models/deepseek-ai/DeepSeek-V2-Lite-Chat...
(VllmWorker rank=0 pid=4011636) INFO 08-21 13:22:07 [model_runner_v1.py:1896] Starting to load model /home/xxx/cache/modelscope/models/deepseek-ai/DeepSeek-V2-Lite-Chat...
[rank1]:[W821 13:22:08.681899385 compiler_depend.ts:2926] Warning: The indexFromRank 0is not equal indexFromCurDevice 1 , which might be normal if the number of devices on your collective communication server is inconsistent.Otherwise, you need to check if the current device is correct when calling the interface.If it's incorrect, it might have introduced an error. (function operator())
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:03<00:10,  3.35s/it]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:07<00:07,  3.77s/it]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:10<00:03,  3.33s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:14<00:00,  3.59s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:14<00:00,  3.55s/it]
(VllmWorker rank=0 pid=4011636) 
(VllmWorker rank=1 pid=4011638) INFO 08-21 13:22:23 [default_loader.py:272] Loading weights took 15.27 seconds
(VllmWorker rank=0 pid=4011636) INFO 08-21 13:22:24 [default_loader.py:272] Loading weights took 14.74 seconds
(VllmWorker rank=1 pid=4011638) INFO 08-21 13:22:25 [model_runner_v1.py:1909] Loading model weights took 14.8519 GB
(VllmWorker rank=0 pid=4011636) INFO 08-21 13:22:25 [model_runner_v1.py:1909] Loading model weights took 14.8519 GB
[rank0]:[W821 13:22:28.409734347 compiler_depend.ts:28] Warning: The oprator of MoeInitRouting will be removed from Pytorch and switch to AscendSpeed after 630. (function operator())
[rank1]:[W821 13:22:28.423089036 compiler_depend.ts:28] Warning: The oprator of MoeInitRouting will be removed from Pytorch and switch to AscendSpeed after 630. (function operator())
(VllmWorker rank=1 pid=4011638) INFO 08-21 13:22:28 [worker_v1.py:193] Available memory: 40933016064, total memory: 65452113920
(VllmWorker rank=0 pid=4011636) INFO 08-21 13:22:29 [worker_v1.py:193] Available memory: 40938664448, total memory: 65452113920
INFO 08-21 13:22:29 [kv_cache_utils.py:715] GPU KV cache size: 1,316,096 tokens
INFO 08-21 13:22:29 [kv_cache_utils.py:719] Maximum concurrency for 16,384 tokens per request: 80.33x
INFO 08-21 13:22:29 [kv_cache_utils.py:715] GPU KV cache size: 1,315,968 tokens
INFO 08-21 13:22:29 [kv_cache_utils.py:719] Maximum concurrency for 16,384 tokens per request: 80.32x
INFO 08-21 13:22:29 [core.py:171] init engine (profile, create kv cache, warmup model) took 4.31 seconds
Adding requests: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 221.99it/s]
Processed prompts:   0%|                                                        | 0/4 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s](VllmWorker rank=1 pid=4011638) /home/xxx/code/vllm-ascend/vllm_ascend/attention/mla_v1.py:835: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
(VllmWorker rank=1 pid=4011638)   seqlen=torch.tensor(
(VllmWorker rank=0 pid=4011636) /home/xxx/code/vllm-ascend/vllm_ascend/attention/mla_v1.py:835: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
(VllmWorker rank=0 pid=4011636)   seqlen=torch.tensor(
Processed prompts: 100%|████████████████████████████████████████████████| 4/4 [00:05<00:00,  1.37s/it, est. speed input: 4.75 toks/s, output: 7.31 toks/s]
Prompt: 'Hello, my name is', Generated text: '***** am a computer expert. My goal is to'
Prompt: 'The president of the United States is', Generated text: ' the head of state and the head of government of'
Prompt: 'The capital of France is', Generated text: ' Paris.\n\nParis is the capital of France'
Prompt: 'The future of AI is', Generated text: ' bright, but it’s not without its challenges'

[Potabk]

DeepSeek W8A8 MTP with V0 Scheduler (A3 DP4 TP4)

OK

#!/bin/sh

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip
nic_name="enp23s0f3"
local_ip="172.22.0.212"

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export VLLM_USE_V1=1
export HCCL_BUFFSIZE=1024

vllm serve /home/cache/modelscope/hub/models/vllm-ascend/DeepSeek-V3-W8A8 \
--host 0.0.0.0 \
--port 8004 \
--data-parallel-size 4 \
--data-parallel-size-local 4 \
--data-parallel-address $local_ip \
--data-parallel-rpc-port 13389 \
--seed 1024 \
--served-model-name deepseek \
--quantization ascend \
--tensor-parallel-size 4 \
--enable-expert-parallel \
--max-num-seqs 8 \
--max-model-len 1024 \
--speculative_config '{"method":"deepseek_mtp","num_speculative_tokens":1}' \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.9 \
--additional-config '{"ascend_scheduler_config":{"enabled":true},"torchair_graph_config":{"enabled":true}}'

{"id":"cmpl-70ece311975f440f8e872ae1af0d7b49","object":"text_completion","created":1755784588,"model":"deepseek","choices":[{"index":0,"text":" at night\n\nSleeping well at night is essential for your overall health and well-being. Here are some tips to help you improve your sleep quality:\n\n### 1. **Establish a Consistent Sleep Schedule**\n   - Go to bed and wake up at the same time every day, even on weekends.\n   - This helps regulate your body's internal clock.\n\n### 2. **Create a Relaxing Bedtime Routine**\n   - Engage in calming activities before bed, such as reading, taking a warm","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":7,"total_tokens":107,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}

[Potabk]

DeepSeek W8A8 MTP with V1 Scheduler (A3 DP4 TP4)

Failed

  #!/bin/sh

  # this obtained through ifconfig
  # nic_name is the network interface name corresponding to local_ip
  nic_name="enp23s0f3"
  local_ip="172.22.0.212"

  export HCCL_IF_IP=$local_ip
  export GLOO_SOCKET_IFNAME=$nic_name
  export TP_SOCKET_IFNAME=$nic_name
  export HCCL_SOCKET_IFNAME=$nic_name
  export OMP_PROC_BIND=false
  export OMP_NUM_THREADS=100
  export VLLM_USE_V1=1
  export HCCL_BUFFSIZE=1024

  vllm serve /home/cache/modelscope/hub/models/vllm-ascend/DeepSeek-V3-W8A8 \
  --host 0.0.0.0 \
  --port 8004 \
  --data-parallel-size 4 \
  --data-parallel-size-local 4 \
  --data-parallel-address $local_ip \
  --data-parallel-rpc-port 13389 \
  --seed 1024 \
  --served-model-name deepseek \
  --quantization ascend \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --max-num-seqs 8 \
  --max-model-len 1024 \
  --speculative_config '{"method":"deepseek_mtp","num_speculative_tokens":1}' \
  --trust-remote-code \
  --no-enable-prefix-caching \
  --gpu-memory-utilization 0.9 \
  --additional-config '{"ascend_scheduler_config":{"enabled":false},"torchair_graph_config":{"enabled":true}}'

 File "/workspace/env/code/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1674, in _dummy_run
(VllmWorker rank=0 pid=58176) ERROR 08-21 14:36:33 [multiproc_executor.py:527]     enable_dbo) = self._get_forward_metadata_across_dp(
(VllmWorker rank=0 pid=58176) ERROR 08-21 14:36:33 [multiproc_executor.py:527]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=58176) ERROR 08-21 14:36:33 [multiproc_executor.py:527]   File "/workspace/env/code/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 649, in _get_forward_metadata_across_dp
(VllmWorker rank=0 pid=58176) ERROR 08-21 14:36:33 [multiproc_executor.py:527]     dist.all_reduce(forward_metadata, group=get_dp_group().cpu_group)
(VllmWorker rank=0 pid=58176) ERROR 08-21 14:36:33 [multiproc_executor.py:527]   File "/workspace/env/091-dev/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
(VllmWorker rank=0 pid=58176) ERROR 08-21 14:36:33 [multiproc_executor.py:527]     return func(*args, **kwargs)
(VllmWorker rank=0 pid=58176) ERROR 08-21 14:36:33 [multiproc_executor.py:527]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=58176) ERROR 08-21 14:36:33 [multiproc_executor.py:527]   File "/workspace/env/091-dev/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 2506, in all_reduce
(VllmWorker rank=0 pid=58176) ERROR 08-21 14:36:33 [multiproc_executor.py:527]     work.wait()
(VllmWorker rank=0 pid=58176) ERROR 08-21 14:36:33 [multiproc_executor.py:527] RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [172.22.0.212]:33158

[Potabk]

DeepSeek W8A8 MTP with V1 Scheduler (A3 DP4 TP4)

After turn down HCCL_BUFFSIZE=512, it seems ok

  #!/bin/sh

  # this obtained through ifconfig
  # nic_name is the network interface name corresponding to local_ip
  nic_name="enp23s0f3"
  local_ip="172.22.0.212"

  export HCCL_IF_IP=$local_ip
  export GLOO_SOCKET_IFNAME=$nic_name
  export TP_SOCKET_IFNAME=$nic_name
  export HCCL_SOCKET_IFNAME=$nic_name
  export OMP_PROC_BIND=false
  export OMP_NUM_THREADS=100
  export VLLM_USE_V1=1
  export HCCL_BUFFSIZE=512

  vllm serve /home/cache/modelscope/hub/models/vllm-ascend/DeepSeek-V3-W8A8 \
  --host 0.0.0.0 \
  --port 8004 \
  --data-parallel-size 4 \
  --data-parallel-size-local 4 \
  --data-parallel-address $local_ip \
  --data-parallel-rpc-port 13389 \
  --seed 1024 \
  --served-model-name deepseek \
  --quantization ascend \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --max-num-seqs 8 \
  --max-model-len 1024 \
  --speculative_config '{"method":"deepseek_mtp","num_speculative_tokens":1}' \
  --trust-remote-code \
  --no-enable-prefix-caching \
  --gpu-memory-utilization 0.9 \
  --additional-config '{"ascend_scheduler_config":{"enabled":false},"torchair_graph_config":{"enabled":true}}'

[zhangxinyuehfad]

✅DeepSeek-R1 W8A8 1P2D + pure DP + lmhead_tensor_parallel_size=8 test OK

command:

cd examples/disaggregate_prefill_v1/

bash gen_ranktable.sh --ips 172.22.0.188 172.22.0.155  \
  --npus-per-node 16 --network-card-name enp23s0f3 --prefill-device-cnt 16 --decode-device-cnt 32
  

prefill

#!/bin/sh

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip
nic_name="enp23s0f3"
local_ip="172.22.0.188"

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export VLLM_USE_V1=1
export HCCL_BUFFSIZE=1024

vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-R1-W8A8 \
--host 0.0.0.0 \
--port 20002 \
--data-parallel-size 2 \
--data-parallel-size-local 2 \
--data-parallel-address $local_ip \
--tensor-parallel-size 8 \
--data-parallel-rpc-port 13389 \
--seed 1024 \
--served-model-name deepseek \
--quantization ascend \
--enable-expert-parallel \
--max-num-seqs 16 \
--max-model-len 2048 \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.9 \
--additional-config '{"torchair_graph_config":{"enabled":true}}'

 

decode-node0

#!/bin/sh

export HCCL_IF_IP=172.22.0.155
export GLOO_SOCKET_IFNAME="enp23s0f3"
export TP_SOCKET_IFNAME="enp23s0f3"
export HCCL_SOCKET_IFNAME="enp23s0f3"
export DISAGGREGATED_PREFILL_RANK_TABLE_PATH=/vllm-workspace/vllm-ascend/examples/disaggregated_prefill_v1/ranktable.json
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export HCCL_BUFFSIZE=600
export VLLM_USE_V1=1
export PYTORCH_NPU_ALLOC_CONF=max_split_size_mb:256

vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-R1-W8A8 \
  --host 0.0.0.0 \
  --port 20002 \
  --data-parallel-size 32 \
  --data-parallel-size-local 16 \
  --data-parallel-address 172.22.0.155 \
  --api-server-count 2 \
  --data-parallel-rpc-port 13389 \
  --served-model-name deepseek \
  --seed 1024 \
  --quantization ascend \
  --enable-expert-parallel \
  --max-num-seqs 16 \
  --max-model-len 2048 \
  --trust-remote-code \
  --no-enable-prefix-caching \
  --gpu-memory-utilization 0.9 \
  --additional-config '{"torchair_graph_config":{"enabled":true},"lmhead_tensor_parallel_size": 8}' &

decode-node1

#!/bin/sh

export HCCL_IF_IP=172.22.0.218
export GLOO_SOCKET_IFNAME="enp23s0f3"
export TP_SOCKET_IFNAME="enp23s0f3"
export HCCL_SOCKET_IFNAME="enp23s0f3"
export DISAGGREGATED_PREFILL_RANK_TABLE_PATH=/vllm-workspace/vllm-ascend/examples/disaggregated_prefill_v1/ranktable.json
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export HCCL_BUFFSIZE=600
export VLLM_USE_V1=1
export PYTORCH_NPU_ALLOC_CONF=max_split_size_mb:256

vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-R1-W8A8 \
  --host 0.0.0.0 \
  --port 20002 \
  --headless \
  --data-parallel-size 32 \
  --data-parallel-start-rank 16 \
  --data-parallel-size-local 16 \
  --data-parallel-address 172.22.0.155 \
  --data-parallel-rpc-port 13389 \
  --served-model-name deepseek \
  --seed 1024 \
  --quantization ascend \
  --enable-expert-parallel \
  --max-num-seqs 16 \
  --max-model-len 2048 \
  --trust-remote-code \
  --no-enable-prefix-caching \
  --gpu-memory-utilization 0.9 \
  --additional-config '{"torchair_graph_config":{"enabled":true},"lmhead_tensor_parallel_size": 8}' &

proxy

python load_balance_proxy_server_example.py --host 0.0.0.0  --port 1025 --prefiller-hosts 172.22.0.188 --prefiller-port 20002 --decoder-hosts 172.22.0.155 --decoder-ports 20002

output：

#!/bin/bash
N=10

for i in $(seq 1 $N); do
  curl http://localhost:1025/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "deepseek",
        "prompt": "The future of AI is",
        "max_tokens": 100,
        "temperature": 0.5
    }'
done

wait

root@liteserver-for-vllm-ascend-00003:/vllm-workspace/vllm-ascend/examples/disaggregate_prefill_v1# ./test.sh
{"id":"cmpl-12","object":"text_completion","created":1755842789,"model":"deepseek","choices":[{"index":0,"text":" here, and it's called ChatGPT. This revolutionary language model is changing the way we interact with technology and each other. In this blog post, we'll explore what ChatGPT is, how it works, and why it's so important. We'll also discuss the potential applications of ChatGPT and how it could shape the future of AI.\\n\\nWhat is ChatGPT?\\n\\nChatGPT is a large language model (LLM) developed by OpenAI. It is a type of AI that can understand and generate","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}{"id":"cmpl-13","object":"text_completion","created":1755842796,"model":"deepseek","choices":[{"index":0,"text":" exciting, but there's a lot of work to do to ensure that these systems are built and deployed responsibly. That's why we need to focus on building trust and transparency into AI systems. Here are some key areas to consider:\n\n- Explainability and Interpretability: AI models, especially complex ones like deep learning, can be \"black boxes\" where it's difficult to understand how they arrive at their decisions. This lack of transparency can lead to distrust. Researchers are working on techniques to make AI models","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}{"id":"cmpl-14","object":"text_completion","created":1755842804,"model":"deepseek","choices":[{"index":0,"text":" not just about technological advancements; it's about how these technologies integrate into our daily lives and reshape industries. As we look ahead, several key trends and predictions emerge that will define the trajectory of artificial intelligence.\n\n1. Democratization of AI\n\nAI tools and platforms are becoming more accessible, enabling businesses of all sizes to leverage AI without needing extensive expertise. Cloud-based AI services, open-source frameworks, and no-code/low-code platforms are lowering the barriers to entry. This democratization will lead to widespread adoption","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}{"id":"cmpl-15","object":"text_completion","created":1755842811,"model":"deepseek","choices":[{"index":0,"text":" here, and it’s called the Apple GPT. Apple has been working on developing its own generative AI model, which is set to rival other existing models like ChatGPT and Google’s Bard. This new model, known as Apple GPT, is based on a framework called “Ajax” and is built on top of Google’s JAX machine learning framework. With this development, Apple aims to create a more advanced and efficient AI system that can be integrated into various applications and services.\n\nThe Apple","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}{"id":"cmpl-16","object":"text_completion","created":1755842818,"model":"deepseek","choices":[{"index":0,"text":" here, and it’s called AutoGPT. This groundbreaking technology is set to revolutionize the way we interact with artificial intelligence, making it more accessible, efficient, and user-friendly than ever before. In this article, we’ll explore what AutoGPT is, how it works, and the potential impact it could have on various industries and our daily lives. So, buckle up and get ready to dive into the world of AutoGPT – the next generation of AI!\n\n## What is AutoGPT?\n\nAuto","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}{"id":"cmpl-17","object":"text_completion","created":1755842826,"model":"deepseek","choices":[{"index":0,"text":" not just about smarter machines; it's about creating a more equitable, sustainable, and human-centered world.\n\nIn the ever-evolving landscape of technology, artificial intelligence (AI) stands out as a transformative force reshaping industries, societies, and daily life. As we look ahead, the future of AI promises both unprecedented opportunities and significant challenges. Understanding where AI is headed requires a deep dive into current trends, potential advancements, ethical considerations, and the societal impacts that will shape our world in the coming decades.\n\n##","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}{"id":"cmpl-18","object":"text_completion","created":1755842833,"model":"deepseek","choices":[{"index":0,"text":" here, and it's called ChatGPT. This revolutionary technology is changing the way we interact with machines and each other. From customer service to content creation, ChatGPT is transforming industries across the board. But with great power comes great responsibility. As we embrace this new era of AI, we must also consider the ethical implications and ensure that ChatGPT is used for the greater good. In this article, we'll explore the power and potential of ChatGPT, its impact on society, and what the future holds for this game","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}{"id":"cmpl-19","object":"text_completion","created":1755842841,"model":"deepseek","choices":[{"index":0,"text":" a topic that has been widely debated and discussed in recent years. As technology continues to advance at an unprecedented rate, the potential applications and implications of artificial intelligence are becoming increasingly significant. From self-driving cars to virtual assistants, AI has already begun to revolutionize various industries and aspects of daily life. However, with this rapid development comes a host of ethical, social, and economic considerations that must be carefully navigated.\n\nOne of the most pressing questions surrounding the future of AI is its impact on the workforce.","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}{"id":"cmpl-20","object":"text_completion","created":1755842848,"model":"deepseek","choices":[{"index":0,"text":" here, and it's changing the way we live and work. From self-driving cars to virtual assistants, AI is already making a big impact on our daily lives. But what does the future hold for this rapidly evolving technology? In this blog post, we'll explore some of the latest developments in AI and what they could mean for the future.\n\nFirst, let's take a look at the current state of AI. Today, AI is being used in a wide range of industries, from healthcare to finance","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}{"id":"cmpl-21","object":"text_completion","created":1755842855,"model":"deepseek","choices":[{"index":0,"text":" not just about technological advancements; it's about creating a future where technology and humanity coexist harmoniously.\n\n# AI and the Future of Work: How Artificial Intelligence is Reshaping Industries and Job Markets\n\nThe rise of artificial intelligence (AI) is transforming the world of work at an unprecedented pace. From automating routine tasks to enabling entirely new industries, AI is reshaping job markets and industries across the globe. While some fear widespread job displacement, others see opportunities for innovation, increased productivity, and the creation of","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}

[dvuiu]

DeepSeek W8A8 MTP with V1 Scheduler (A3 DP4 TP4)

After turn down HCCL_BUFFSIZE=512, it seems ok

#!/bin/sh

this obtained through ifconfig

nic_name is the network interface name corresponding to local_ip

nic_name="enp23s0f3"
local_ip="172.22.0.212"

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export VLLM_USE_V1=1
export HCCL_BUFFSIZE=512

vllm serve /home/cache/modelscope/hub/models/vllm-ascend/DeepSeek-V3-W8A8
--host 0.0.0.0
--port 8004
--data-parallel-size 4
--data-parallel-size-local 4
--data-parallel-address $local_ip
--data-parallel-rpc-port 13389
--seed 1024
--served-model-name deepseek
--quantization ascend
--tensor-parallel-size 4
--enable-expert-parallel
--max-num-seqs 8
--max-model-len 1024
--speculative_config '{"method":"deepseek_mtp","num_speculative_tokens":1}'
--trust-remote-code
--no-enable-prefix-caching
--gpu-memory-utilization 0.9
--additional-config '{"ascend_scheduler_config":{"enabled":false},"torchair_graph_config":{"enabled":true}}'

环境：A2, 264G8,
镜像：docker pull quay.io/ascend/vllm-ascend:v0.9.1rc3

出现如下报错：
(VllmWorker rank=0 pid=1275) INFO 08-28 07:44:49 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_3662e765'), local_subscribe_addr='ipc:///tmp/d4572351-53f1-494a-a8fa-03153f82bfa1', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 08-28 07:44:49 [utils.py:2737] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes not implemented in <vllm_ascend.worker.worker_v1.NPUWorker object at 0xfffdbf65b050>
(VllmWorker rank=0 pid=1272) INFO 08-28 07:44:49 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_4a6595b2'), local_subscribe_addr='ipc:///tmp/a0e43aa0-47cd-4deb-91a4-abf1fc328614', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 08-28 07:44:49 [utils.py:2737] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes not implemented in <vllm_ascend.worker.worker_v1.NPUWorker object at 0xffff857d3d90>
(VllmWorker rank=0 pid=1274) INFO 08-28 07:44:49 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_d873b103'), local_subscribe_addr='ipc:///tmp/449e22d8-1258-403f-b5b3-794fbc9e834e', remote_subscribe_addr=None, remote_addr_ipv6=False)
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] WorkerProc failed to start.
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] Traceback (most recent call last):
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 466, in worker_main
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] worker = WorkerProc(*args, **kwargs)
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 362, in init
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] self.worker.init_device()
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] File "/vllm-workspace/vllm/vllm/worker/worker_base.py", line 606, in init_device
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] self.worker.init_device() # type: ignore
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] ^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 140, in init_device
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] NPUPlatform.set_device(self.device)
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] File "/vllm-workspace/vllm-ascend/vllm_ascend/platform.py", line 100, in set_device
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] torch.npu.set_device(device)
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/npu/utils.py", line 80, in set_device
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] torch_npu._C._npu_setDevice(device_id)
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] RuntimeError: init:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:292 NPU function error: aclrtGetDeviceCount(&device_count), error code is 207004
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] [ERROR] 2025-08-28-07:44:50 (PID:1273, Device:-1, RankID:-1) ERR00100 PTA call acl api failed
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] [Error]: The device is unavailable.
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] Check whether the device is running properly.
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] EE1001: [PID: 1273] 2025-08-28-07:44:50.179.176 The argument is invalid.Reason: set ASCEND_RT_VISIBLE_DEVICES:12,13,14,15 error, input data rang[0-8)
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] TraceBack (most recent call last):
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] rtGetDeviceCount execute failed, reason=[driver error:no valid device][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] get device count failed, runtime result = 207004.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] The argument is invalid.Reason: Set visible device failed, invalid device=0, input visible devices:12,13,14,15
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] rtSetDevice execute failed, reason=[device id error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] open device 0 failed, runtime result = 107001.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6147]
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492] The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]
(VllmWorker rank=0 pid=1273) ERROR 08-28 07:44:50 [multiproc_executor.py:492]
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] WorkerProc failed to start.
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] Traceback (most recent call last):
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 466, in worker_main
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] worker = WorkerProc(*args, **kwargs)
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 362, in init
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] self.worker.init_device()
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] File "/vllm-workspace/vllm/vllm/worker/worker_base.py", line 606, in init_device
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] self.worker.init_device() # type: ignore
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] ^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 140, in init_device
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] NPUPlatform.set_device(self.device)
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] File "/vllm-workspace/vllm-ascend/vllm_ascend/platform.py", line 100, in set_device
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] torch.npu.set_device(device)
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/npu/utils.py", line 80, in set_device
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] torch_npu._C._npu_setDevice(device_id)
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] RuntimeError: init:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:292 NPU function error: aclrtGetDeviceCount(&device_count), error code is 207004
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] [ERROR] 2025-08-28-07:44:50 (PID:1274, Device:-1, RankID:-1) ERR00100 PTA call acl api failed
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] [Error]: The device is unavailable.
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] Check whether the device is running properly.
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] EE1001: [PID: 1274] 2025-08-28-07:44:50.751.957 The argument is invalid.Reason: set ASCEND_RT_VISIBLE_DEVICES:8,9,10,11 error, input data rang[0-8)
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] TraceBack (most recent call last):
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] rtGetDeviceCount execute failed, reason=[driver error:no valid device][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] get device count failed, runtime result = 207004.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] The argument is invalid.Reason: Set visible device failed, invalid device=0, input visible devices:8,9,10,11
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] rtSetDevice execute failed, reason=[device id error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] open device 0 failed, runtime result = 107001.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6147]
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492] The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]
(VllmWorker rank=0 pid=1274) ERROR 08-28 07:44:50 [multiproc_executor.py:492]