[Release]: Release checklist for v0.11.0rc1

[wangxiyuan]

Release Checklist

Release Version: v0.11.0rc1
Release Branch: main
Release Date: during 0924-0928
Release Manager: @wangxiyuan

Prepare Release Note

• Create a new issue for release feedback [v0.11.0rc1] FAQ / Feedback | 问题/反馈 #3222

• Upgrade vllm version to the new version for CI and Dockerfile Bump version to v0.11.0rc3 #3213

• Write the release note PR. [Doc] Release note for v0.11.0rc0 #3224

  • Update the feedback issue link in docs/source/faqs.md

  • Add release note to docs/source/user_guide/release_notes.md

  • Update release version in README.md and README.zh.md

  • Update version info in docs/source/community/versioning_policy.md

  • Update contributor info in docs/source/community/contributors.md

  • Update package version in docs/conf.py

PR need Merge

• [BugFix] Fix aclgraph accu problem in A2. #3163
• [Bugfix][LoRA] Fix forward error and shape mismatch when using LoRA #3153
• [BugFix] Fix ascend scheduler assert error #3191
• [3/N][Refactor][Qwen3-Next] Refacotr model structure and fix bug by vllm #25400  #3142
• [Eagle] Fix attn_mask index out of range in high concurrency situations #3187
• [Bugfix]Fix quant_config input parameter bug in qwenvl series #3220
• [BugFix] Fix ACLgraph bug in Qwen3_32b_int8 case #3204
• [CI][Bugfix] Quickfix for DPMetaData #3234
• [Doc]Add qwen3_vl series guide #3227
• [Model] Optimizing gemma3 model's GemmaRMSNorm function #3151
• [bugfix] Fix warning bug: model config is None. #3238
• [BugFix] Fix Qwen3-Next because of vllm #24982 #3221

Functional Test

• aclgraph + qwen3 235b @Potabk
• aclgraph + full graph + qwen3 30b @zhangxinyuehfad
• glm4.5 @shen-shanshan
• Qwen3-next @wxsIcey
• Qwen3-vl @booker123456
• Longcat @zhangxinyuehfad

Doc Test

• Tutorial is updated.
  • qwen3-vl guide @booker123456 [Doc]Add qwen3_vl series guide #3227
  • deepseek v3.2 guide [Doc] Add deepseek v3.2 tutorial #3275
• User Guide is updated.
• Developer Guide is updated.

Prepare Artifacts

• Docker image is ready.
• Wheel package is ready.

Release Step

• Release note PR is merged.
• Post the release on GitHub release page.
• Generate official doc page on https://app.readthedocs.org/dashboard/
• Wait for the wheel package to be available on https://pypi.org/project/vllm-ascend
• Wait for the docker image to be available on https://quay.io/ascend/vllm-ascend
• Upload 310p wheel to Github release page
• Broadcast the release news (By message, blog , etc)
• Close this issue

[Potabk]

test Qwen3-235B-A22B with tp 16

vllm serve /home/cache/modelscope/hub/models/Qwen/Qwen3-235B-A22B \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 16 \
  --enable-expert-parallel \
  --seed 1024 \
  --max-model-len 8192  \
  --max-num-seqs 25 \
  --trust-remote-code \
  --gpu-memory-utilization 0.9

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "/home/cache/modelscope/hub/models/Qwen/Qwen3-235B-A22B",
        "prompt": "tell me how to sleep well",
        "max_tokens": 100,
        "temperature": 0
    }'

result is ok:

{"id":"cmpl-8eae97b6a9e148d0a95ca90241d7ca0f","object":"text_completion","created":1759040074,"model":"/home/cache/modelscope/hub/models/Qwen/Qwen3-235B-A22B","choices":[{"index":0,"text":"\nImproving sleep quality involves a combination of good sleep hygiene, lifestyle adjustments, and creating a conducive environment. Here's a comprehensive guide to help you sleep better:\n\n---\n\n### **1. Stick to a Consistent Sleep Schedule**\n- **Go to bed and wake up at the same time every day**, even on weekends. This helps regulate your body's internal clock (circadian rhythm).\n- Aim for **7–9 hours of sleep** per night (adults).\n\n---\n\n### **2.","logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null,"prompt_logprobs":null,"prompt_token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}

test with dp2 tp8

export HCCL_IF_IP=172.17.0.3  # node ip
export GLOO_SOCKET_IFNAME="eth0"  # network card name
export TP_SOCKET_IFNAME="eth0"
export HCCL_SOCKET_IFNAME="eth0"
export OMP_PROC_BIND=false
export HCCL_BUFFSIZE=1024
export OMP_NUM_THREADS=100

vllm serve /home/cache/modelscope/hub/models/Qwen/Qwen3-235B-A22B \
  --host 0.0.0.0 \
  --port 20002 \
  --data-parallel-size 2 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --seed 1024 \
  --enforce-eager \
  --max-model-len 8192  \
  --max-num-seqs 125 \
  --trust-remote-code \
  --gpu-memory-utilization 0.9

curl http://localhost:20002/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "/home/cache/modelscope/hub/models/Qwen/Qwen3-235B-A22B",
        "prompt": "tell me how to sleep well",
        "max_tokens": 100,
        "temperature": 0
    }'

result can be normal:

{"id":"cmpl-5460570d315547a3aa58276b52776d0c","object":"text_completion","created":1759040450,"model":"/home/cache/modelscope/hub/models/Qwen/Qwen3-235B-A22B","choices":[{"index":0,"text":"\nImproving sleep quality involves a combination of good sleep hygiene, lifestyle adjustments, and creating a conducive environment. Here's a comprehensive guide to help you sleep better:\n\n---\n\n### **1. Establish a Consistent Sleep Schedule**\n- **Go to bed and wake up at the same time every day**, even on weekends. This helps regulate your body's internal clock (circadian rhythm).\n- Aim for **7–9 hours of sleep** per night (varies by age and individual needs).\n\n---\n\n","logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null,"prompt_logprobs":null,"prompt_token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}

but aclgraph is not valid:

(Worker_DP0_TP0_EP0 pid=563360) WARNING 09-28 06:20:10 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.

[zhangxinyuehfad]

❌test new model meituan-longcat/LongCat-Flash-Chat failed:

command:

vllm serve /root/.cache/modelscope/hub/models/meituan-longcat/LongCat-Flash-Chat \
--tensor-parallel-size 16 \
--enable-expert-parallel \
--max-num-seqs 16 \
--max-model-len 4096 \
--trust-remote-code \
--served-model-name LongCat \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.7 &

error log:

(Worker_TP0_EP0 pid=546270) INFO 09-28 06:06:29 [model_runner_v1.py:2626] Starting to load model /root/.cache/modelscope/hub/models/meituan-longcat/LongCat-Flash-Chat...
(Worker_TP10_EP10 pid=546280) INFO 09-28 06:06:29 [model_runner_v1.py:2626] Starting to load model /root/.cache/modelscope/hub/models/meituan-longcat/LongCat-Flash-Chat...
(Worker_TP2_EP2 pid=546272) INFO 09-28 06:06:29 [model_runner_v1.py:2626] Starting to load model /root/.cache/modelscope/hub/models/meituan-longcat/LongCat-Flash-Chat...
(Worker_TP3_EP3 pid=546273) INFO 09-28 06:06:29 [model_runner_v1.py:2626] Starting to load model /root/.cache/modelscope/hub/models/meituan-longcat/LongCat-Flash-Chat...
(Worker_TP6_EP6 pid=546276) INFO 09-28 06:06:29 [model_runner_v1.py:2626] Starting to load model /root/.cache/modelscope/hub/models/meituan-longcat/LongCat-Flash-Chat...
(Worker_TP4_EP4 pid=546274) INFO 09-28 06:06:29 [model_runner_v1.py:2626] Starting to load model /root/.cache/modelscope/hub/models/meituan-longcat/LongCat-Flash-Chat...
(Worker_TP15_EP15 pid=546285) INFO 09-28 06:06:29 [model_runner_v1.py:2626] Starting to load model /root/.cache/modelscope/hub/models/meituan-longcat/LongCat-Flash-Chat...
(Worker_TP1_EP1 pid=546271) INFO 09-28 06:06:29 [model_runner_v1.py:2626] Starting to load model /root/.cache/modelscope/hub/models/meituan-longcat/LongCat-Flash-Chat...
(Worker_TP14_EP14 pid=546284) INFO 09-28 06:06:29 [model_runner_v1.py:2626] Starting to load model /root/.cache/modelscope/hub/models/meituan-longcat/LongCat-Flash-Chat...
(Worker_TP8_EP8 pid=546278) INFO 09-28 06:06:29 [model_runner_v1.py:2626] Starting to load model /root/.cache/modelscope/hub/models/meituan-longcat/LongCat-Flash-Chat...
(Worker_TP13_EP13 pid=546283) INFO 09-28 06:06:29 [model_runner_v1.py:2626] Starting to load model /root/.cache/modelscope/hub/models/meituan-longcat/LongCat-Flash-Chat...
(Worker_TP5_EP5 pid=546275) INFO 09-28 06:06:29 [model_runner_v1.py:2626] Starting to load model /root/.cache/modelscope/hub/models/meituan-longcat/LongCat-Flash-Chat...
(Worker_TP7_EP7 pid=546277) INFO 09-28 06:06:29 [model_runner_v1.py:2626] Starting to load model /root/.cache/modelscope/hub/models/meituan-longcat/LongCat-Flash-Chat...
(Worker_TP9_EP9 pid=546279) INFO 09-28 06:06:29 [model_runner_v1.py:2626] Starting to load model /root/.cache/modelscope/hub/models/meituan-longcat/LongCat-Flash-Chat...
(Worker_TP11_EP11 pid=546281) INFO 09-28 06:06:29 [model_runner_v1.py:2626] Starting to load model /root/.cache/modelscope/hub/models/meituan-longcat/LongCat-Flash-Chat...
(Worker_TP12_EP12 pid=546282) INFO 09-28 06:06:29 [model_runner_v1.py:2626] Starting to load model /root/.cache/modelscope/hub/models/meituan-longcat/LongCat-Flash-Chat...
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597] WorkerProc failed to start.
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597] Traceback (most recent call last):
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 571, in worker_main
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]     worker = WorkerProc(*args, **kwargs)
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 437, in __init__
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]     self.worker.load_model()
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 280, in load_model
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]     self.model_runner.load_model()
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2629, in load_model
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]     self.model = get_model(vllm_config=self.vllm_config)
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/__init__.py", line 119, in get_model
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]     return loader.load_model(vllm_config=vllm_config,
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/base_loader.py", line 45, in load_model
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]     model = initialize_model(vllm_config=vllm_config,
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/model_executor/model_loader/utils.py", line 63, in initialize_model
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]     return model_class(vllm_config=vllm_config, prefix=prefix)
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/model_executor/models/longcat_flash.py", line 537, in __init__
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]     self.model = FlashModel(vllm_config=vllm_config,
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/compilation/decorators.py", line 201, in __init__
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]     old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/model_executor/models/longcat_flash.py", line 454, in __init__
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]     self.start_layer, self.end_layer, self.layers = make_layers(
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]                                                     ^^^^^^^^^^^^
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/model_executor/models/utils.py", line 626, in make_layers
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]     [PPMissingLayer() for _ in range(start_layer)] + [
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]                                                      ^
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/model_executor/models/utils.py", line 627, in <listcomp>
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]     maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/model_executor/models/longcat_flash.py", line 456, in <lambda>
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]     lambda prefix: FlashDecoderLayer(
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]                    ^^^^^^^^^^^^^^^^^^
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/model_executor/models/longcat_flash.py", line 330, in __init__
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]     self.self_attn = nn.ModuleList([
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]                                    ^
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/model_executor/models/longcat_flash.py", line 331, in <listcomp>
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]     DeepseekV2MLAAttention(
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]   File "/vllm-workspace/vllm/vllm/model_executor/models/deepseek_v2.py", line 592, in __init__
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]     self.mla_attn = MultiHeadLatentAttention(
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597]                     ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP10_EP10 pid=546280) ERROR 09-28 06:06:30 [multiproc_executor.py:597] TypeError: AscendMultiHeadLatentAttention.__init__() missing 3 required positional arguments: 'qk_nope_head_dim', 'qk_head_dim', and 'v_head_dim'
(Worker_TP10_EP10 pid=546280) INFO 09-28 06:06:30 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP8_EP8 pid=546278) INFO 09-28 06:06:30 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP12_EP12 pid=546282) INFO 09-28 06:06:30 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP0_EP0 pid=546270) INFO 09-28 06:06:30 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP9_EP9 pid=546279) INFO 09-28 06:06:30 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP7_EP7 pid=546277) INFO 09-28 06:06:30 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP6_EP6 pid=546276) INFO 09-28 06:06:30 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP14_EP14 pid=546284) INFO 09-28 06:06:30 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP2_EP2 pid=546272) INFO 09-28 06:06:30 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP4_EP4 pid=546274) INFO 09-28 06:06:30 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP3_EP3 pid=546273) INFO 09-28 06:06:30 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP11_EP11 pid=546281) INFO 09-28 06:06:30 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP5_EP5 pid=546275) INFO 09-28 06:06:30 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP1_EP1 pid=546271) INFO 09-28 06:06:30 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP15_EP15 pid=546285) INFO 09-28 06:06:30 [multiproc_executor.py:558] Parent process exited, terminating worker
(Worker_TP13_EP13 pid=546283) INFO 09-28 06:06:30 [multiproc_executor.py:558] Parent process exited, terminating worker
(EngineCore_DP0 pid=546115) ERROR 09-28 06:06:34 [core.py:708] EngineCore failed to start.
(EngineCore_DP0 pid=546115) ERROR 09-28 06:06:34 [core.py:708] Traceback (most recent call last):
(EngineCore_DP0 pid=546115) ERROR 09-28 06:06:34 [core.py:708]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 699, in run_engine_core
(EngineCore_DP0 pid=546115) ERROR 09-28 06:06:34 [core.py:708]     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=546115) ERROR 09-28 06:06:34 [core.py:708]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=546115) ERROR 09-28 06:06:34 [core.py:708]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 498, in __init__
(EngineCore_DP0 pid=546115) ERROR 09-28 06:06:34 [core.py:708]     super().__init__(vllm_config, executor_class, log_stats,
(EngineCore_DP0 pid=546115) ERROR 09-28 06:06:34 [core.py:708]   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 83, in __init__
(EngineCore_DP0 pid=546115) ERROR 09-28 06:06:34 [core.py:708]     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=546115) ERROR 09-28 06:06:34 [core.py:708]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=546115) ERROR 09-28 06:06:34 [core.py:708]   File "/vllm-workspace/vllm/vllm/executor/executor_base.py", line 54, in __init__
(EngineCore_DP0 pid=546115) ERROR 09-28 06:06:34 [core.py:708]     self._init_executor()
(EngineCore_DP0 pid=546115) ERROR 09-28 06:06:34 [core.py:708]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 106, in _init_executor
(EngineCore_DP0 pid=546115) ERROR 09-28 06:06:34 [core.py:708]     self.workers = WorkerProc.wait_for_ready(unready_workers)
(EngineCore_DP0 pid=546115) ERROR 09-28 06:06:34 [core.py:708]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=546115) ERROR 09-28 06:06:34 [core.py:708]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 509, in wait_for_ready
(EngineCore_DP0 pid=546115) ERROR 09-28 06:06:34 [core.py:708]     raise e from None
(EngineCore_DP0 pid=546115) ERROR 09-28 06:06:34 [core.py:708] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
(EngineCore_DP0 pid=546115) Process EngineCore_DP0:
(EngineCore_DP0 pid=546115) Traceback (most recent call last):
(EngineCore_DP0 pid=546115)   File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=546115)     self.run()
(EngineCore_DP0 pid=546115)   File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=546115)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=546115)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 712, in run_engine_core
(EngineCore_DP0 pid=546115)     raise e
(EngineCore_DP0 pid=546115)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 699, in run_engine_core
(EngineCore_DP0 pid=546115)     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=546115)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=546115)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 498, in __init__
(EngineCore_DP0 pid=546115)     super().__init__(vllm_config, executor_class, log_stats,
(EngineCore_DP0 pid=546115)   File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 83, in __init__
(EngineCore_DP0 pid=546115)     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=546115)                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=546115)   File "/vllm-workspace/vllm/vllm/executor/executor_base.py", line 54, in __init__
(EngineCore_DP0 pid=546115)     self._init_executor()
(EngineCore_DP0 pid=546115)   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 106, in _init_executor
(EngineCore_DP0 pid=546115)     self.workers = WorkerProc.wait_for_ready(unready_workers)
(EngineCore_DP0 pid=546115)                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=546115)   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 509, in wait_for_ready
(EngineCore_DP0 pid=546115)     raise e from None
(EngineCore_DP0 pid=546115) Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
(APIServer pid=545828) Traceback (most recent call last):
(APIServer pid=545828)   File "/usr/local/python3.11.13/bin/vllm", line 8, in <module>
(APIServer pid=545828)     sys.exit(main())
(APIServer pid=545828)              ^^^^^^
(APIServer pid=545828)   File "/vllm-workspace/vllm/vllm/entrypoints/cli/main.py", line 54, in main
(APIServer pid=545828)     args.dispatch_function(args)
(APIServer pid=545828)   File "/vllm-workspace/vllm/vllm/entrypoints/cli/serve.py", line 57, in cmd
(APIServer pid=545828)     uvloop.run(run_server(args))
(APIServer pid=545828)   File "/usr/local/python3.11.13/lib/python3.11/site-packages/uvloop/__init__.py", line 105, in run
(APIServer pid=545828)     return runner.run(wrapper())
(APIServer pid=545828)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=545828)   File "/usr/local/python3.11.13/lib/python3.11/asyncio/runners.py", line 118, in run
(APIServer pid=545828)     return self._loop.run_until_complete(task)
(APIServer pid=545828)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=545828)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=545828)   File "/usr/local/python3.11.13/lib/python3.11/site-packages/uvloop/__init__.py", line 61, in wrapper
(APIServer pid=545828)     return await main
(APIServer pid=545828)            ^^^^^^^^^^
(APIServer pid=545828)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 1863, in run_server
(APIServer pid=545828)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=545828)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 1881, in run_server_worker
(APIServer pid=545828)     async with build_async_engine_client(
(APIServer pid=545828)   File "/usr/local/python3.11.13/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=545828)     return await anext(self.gen)
(APIServer pid=545828)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=545828)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 178, in build_async_engine_client
(APIServer pid=545828)     async with build_async_engine_client_from_engine_args(
(APIServer pid=545828)   File "/usr/local/python3.11.13/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=545828)     return await anext(self.gen)
(APIServer pid=545828)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=545828)   File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 223, in build_async_engine_client_from_engine_args
(APIServer pid=545828)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=545828)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=545828)   File "/vllm-workspace/vllm/vllm/utils/__init__.py", line 1571, in inner
(APIServer pid=545828)     return fn(*args, **kwargs)
(APIServer pid=545828)            ^^^^^^^^^^^^^^^^^^^
(APIServer pid=545828)   File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 207, in from_vllm_config
(APIServer pid=545828)     return cls(
(APIServer pid=545828)            ^^^^
(APIServer pid=545828)   File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 134, in __init__
(APIServer pid=545828)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=545828)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=545828)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 102, in make_async_mp_client
(APIServer pid=545828)     return AsyncMPClient(*client_args)
(APIServer pid=545828)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=545828)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 769, in __init__
(APIServer pid=545828)     super().__init__(
(APIServer pid=545828)   File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 448, in __init__
(APIServer pid=545828)     with launch_core_engines(vllm_config, executor_class,
(APIServer pid=545828)   File "/usr/local/python3.11.13/lib/python3.11/contextlib.py", line 144, in __exit__
(APIServer pid=545828)     next(self.gen)
(APIServer pid=545828)   File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 732, in launch_core_engines
(APIServer pid=545828)     wait_for_engine_startup(
(APIServer pid=545828)   File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 785, in wait_for_engine_startup
(APIServer pid=545828)     raise RuntimeError("Engine core initialization failed. "
(APIServer pid=545828) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(APIServer pid=545828) [ERROR] 2025-09-28-06:06:36 (PID:545828, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
/usr/local/python3.11.13/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

[zhangxinyuehfad]

✅ test qwen3 30b + aclgraph + full grap OK

command:

vllm serve /root/.cache/modelscope/hub/models/Qwen/Qwen3-30B-A3B \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --trust-remote-code \
  --served-model-name qwen_30B \
  --gpu-memory-utilization 0.7 \
  --compilation-config '{"level": 3,"cudagraph_mode": "FULL_DECODE_ONLY"}' &

log

Capturing ACL graphs (decode, FULL):   0%|                                                                                                                                                                                                                             | 0/35 [00:00<?, ?it/s](Worker_TP2_EP2 pid=729699) WARNING 09-28 07:25:40 [__init__.py:755] Current vLLM config is not set.
(Worker_TP3_EP3 pid=729700) WARNING 09-28 07:25:40 [__init__.py:755] Current vLLM config is not set.
(Worker_TP0_EP0 pid=729697) WARNING 09-28 07:25:40 [__init__.py:755] Current vLLM config is not set.
(Worker_TP1_EP1 pid=729698) WARNING 09-28 07:25:40 [__init__.py:755] Current vLLM config is not set.
(Worker_TP2_EP2 pid=729699) WARNING 09-28 07:25:40 [platform.py:169] Model config is missing. This may indicate that we are running a test case
(Worker_TP3_EP3 pid=729700) WARNING 09-28 07:25:40 [platform.py:169] Model config is missing. This may indicate that we are running a test case
(Worker_TP0_EP0 pid=729697) WARNING 09-28 07:25:40 [platform.py:169] Model config is missing. This may indicate that we are running a test case
(Worker_TP1_EP1 pid=729698) WARNING 09-28 07:25:40 [platform.py:169] Model config is missing. This may indicate that we are running a test case
[rank2]:[W928 07:25:40.530834161 compiler_depend.ts:149] Warning: Waiting for pending NCCL work to finish before starting graph capture. (function operator())
[rank0]:[W928 07:25:40.564340887 compiler_depend.ts:149] Warning: Waiting for pending NCCL work to finish before starting graph capture. (function operator())
[rank3]:[W928 07:25:40.576471032 compiler_depend.ts:149] Warning: Waiting for pending NCCL work to finish before starting graph capture. (function operator())
[rank1]:[W928 07:25:41.609955727 compiler_depend.ts:149] Warning: Waiting for pending NCCL work to finish before starting graph capture. (function operator())
Capturing ACL graphs (decode, FULL):  43%|██████████████████████████████████████████████████████████████████████████████████████████▊                                                                                                                         | 15/35 [00:56<01:44,  5.22s/it](EngineCore_DP0 pid=729545) INFO 09-28 07:26:40 [shm_broadcast.py:466] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation).
Capturing ACL graphs (decode, FULL):  66%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎                                                                        | 23/35 [01:52<01:26,  7.23s/it](EngineCore_DP0 pid=729545) INFO 09-28 07:27:40 [shm_broadcast.py:466] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation).
Capturing ACL graphs (decode, FULL):  86%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋                              | 30/35 [02:57<00:47,  9.54s/it](EngineCore_DP0 pid=729545) INFO 09-28 07:28:40 [shm_broadcast.py:466] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation).
Capturing ACL graphs (decode, FULL):  97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉      | 34/35 [03:40<00:10, 10.58s/it](Worker_TP2_EP2 pid=729699) INFO 09-28 07:29:32 [model_runner_v1.py:3462] Graph capturing finished in 232 secs, took 1.35 GiB
(Worker_TP3_EP3 pid=729700) INFO 09-28 07:29:32 [model_runner_v1.py:3462] Graph capturing finished in 232 secs, took 1.35 GiB
(Worker_TP1_EP1 pid=729698) INFO 09-28 07:29:32 [model_runner_v1.py:3462] Graph capturing finished in 232 secs, took 1.35 GiB
Capturing ACL graphs (decode, FULL): 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [03:52<00:00,  6.63s/it]
(Worker_TP0_EP0 pid=729697) INFO 09-28 07:29:32 [model_runner_v1.py:3462] Graph capturing finished in 232 secs, took 1.35 GiB
(EngineCore_DP0 pid=729545) INFO 09-28 07:29:32 [core.py:210] init engine (profile, create kv cache, warmup model) took 249.17 seconds
(EngineCore_DP0 pid=729545) WARNING 09-28 07:29:33 [core.py:112] Using configured V1 scheduler class vllm_ascend.core.scheduler.AscendScheduler. This scheduler interface is not public and compatibility may not be maintained.
(EngineCore_DP0 pid=729545) INFO 09-28 07:29:33 [platform.py:140] Non-MLA LLMs forcibly disable the chunked prefill feature,as the performance of operators supporting this feature functionality is currently suboptimal.
(EngineCore_DP0 pid=729545) INFO 09-28 07:29:33 [platform.py:238] FULL_DECODE_ONLY compilation enabled on NPU. use_inductor not supported - using only ACL Graph mode
(EngineCore_DP0 pid=729545) WARNING 09-28 07:29:33 [platform.py:254]
(EngineCore_DP0 pid=729545) WARNING 09-28 07:29:33 [platform.py:254]             **********************************************************************************
(EngineCore_DP0 pid=729545) WARNING 09-28 07:29:33 [platform.py:254]             * WARNING: You have enabled the *full graph* feature.
(EngineCore_DP0 pid=729545) WARNING 09-28 07:29:33 [platform.py:254]             * This is an early experimental stage and may involve various unknown issues.
(EngineCore_DP0 pid=729545) WARNING 09-28 07:29:33 [platform.py:254]             * A known problem is that capturing too many batch sizes can lead to OOM
(EngineCore_DP0 pid=729545) WARNING 09-28 07:29:33 [platform.py:254]             * (Out of Memory) errors or inference hangs. If you encounter such issues,
(EngineCore_DP0 pid=729545) WARNING 09-28 07:29:33 [platform.py:254]             * consider reducing `gpu_memory_utilization` or manually specifying a smaller
(EngineCore_DP0 pid=729545) WARNING 09-28 07:29:33 [platform.py:254]             * batch size for graph capture.
(EngineCore_DP0 pid=729545) WARNING 09-28 07:29:33 [platform.py:254]             * For more details, please refer to:
(EngineCore_DP0 pid=729545) WARNING 09-28 07:29:33 [platform.py:254]             * https://docs.vllm.ai/en/stable/configuration/conserving_memory.html#reduce-cuda-graphs
(EngineCore_DP0 pid=729545) WARNING 09-28 07:29:33 [platform.py:254]             **********************************************************************************
(EngineCore_DP0 pid=729545) WARNING 09-28 07:29:33 [platform.py:254]
(APIServer pid=729110) INFO 09-28 07:29:33 [loggers.py:147] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 7905
(APIServer pid=729110) INFO 09-28 07:29:33 [api_server.py:1614] Supported_tasks: ['generate']
(APIServer pid=729110) WARNING 09-28 07:29:33 [model.py:1386] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=729110) INFO 09-28 07:29:33 [serving_responses.py:137] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
(APIServer pid=729110) INFO 09-28 07:29:33 [serving_chat.py:137] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
(APIServer pid=729110) INFO 09-28 07:29:33 [serving_completion.py:76] Using default completion sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
(APIServer pid=729110) INFO 09-28 07:29:33 [api_server.py:1891] Starting vLLM API server 0 on http://0.0.0.0:8000
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:34] Available routes are:
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /docs, Methods: HEAD, GET
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /redoc, Methods: HEAD, GET
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /health, Methods: GET
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /load, Methods: GET
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /ping, Methods: POST
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /ping, Methods: GET
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /tokenize, Methods: POST
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /detokenize, Methods: POST
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /v1/models, Methods: GET
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /version, Methods: GET
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /v1/responses, Methods: POST
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /v1/chat/completions, Methods: POST
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /v1/completions, Methods: POST
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /v1/embeddings, Methods: POST
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /pooling, Methods: POST
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /classify, Methods: POST
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /score, Methods: POST
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /v1/score, Methods: POST
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /v1/audio/translations, Methods: POST
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /rerank, Methods: POST
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /v1/rerank, Methods: POST
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /v2/rerank, Methods: POST
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /invocations, Methods: POST
(APIServer pid=729110) INFO 09-28 07:29:33 [launcher.py:42] Route: /metrics, Methods: GET
(APIServer pid=729110) INFO:     Started server process [729110]
(APIServer pid=729110) INFO:     Waiting for application startup.
(APIServer pid=729110) INFO:     Application startup complete.
(Worker_TP1_EP1 pid=729698) INFO 09-28 07:29:54 [acl_graph.py:187] Replaying aclgraph
(Worker_TP0_EP0 pid=729697) INFO 09-28 07:29:54 [acl_graph.py:187] Replaying aclgraph
(Worker_TP2_EP2 pid=729699) INFO 09-28 07:29:54 [acl_graph.py:187] Replaying aclgraph
(Worker_TP3_EP3 pid=729700) INFO 09-28 07:29:54 [acl_graph.py:187] Replaying aclgraph
(APIServer pid=729110) INFO:     127.0.0.1:35832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=729110) INFO 09-28 07:30:03 [loggers.py:127] Engine 000: Avg prompt throughput: 0.4 tokens/s, Avg generation throughput: 10.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%

output:

root@liteserver-for-vllm-ascend-00003:/vllm-workspace/vllm-ascend# curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "qwen_30B",
        "prompt": "西安有什么好玩的地方",
        "max_tokens": 100,
        "temperature": 0
    }'
{"id":"cmpl-cd93d5ca72c54c8b82550cba3a86696a","object":"text_completion","created":1759044594,"model":"qwen_30B","choices":[{"index":0,"text":"？西安有哪些值得去的景点？ 西安有哪些值得去的景点？ 西安有哪些值得去的景点？\n\n西安是中国历史文化名城，拥有众多著名景点。以下是一些值得去的景点：\n\n1. 兵马俑：世界第八大奇迹，位于临潼区，是秦始皇陵的陪葬坑，展示了秦朝军队的壮观景象。\n\n2. 大雁塔：位于大慈恩寺内，是玄奘法师藏经","logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null,"prompt_logprobs":null,"prompt_token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":4,"total_tokens":104,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}root@liteserver-for-vllm-ascend-00003:/vllm-workspace/vllm-ascend#

[shen-shanshan]

GLM-4.5 Test

Preparation

Pull docker:

docker pull quay.nju.edu.cn/ascend/vllm-ascend:main-a3

Config env:

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export VLLM_USE_V1=1
export VLLM_WORKER_MULTIPROC_METHOD="spawn"
export VLLM_USE_MODELSCOPE=True
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export HCCL_BUFFSIZE=2048
export PYTORCH_NPU_ALLOC_CONF=max_split_size_mb:256

✅ 1. Eager Mode (TP16+EP & Single Node)

Run:

vllm serve /root/.cache/modelscope/hub/models/ZhipuAI/GLM-4___5 \
--host 0.0.0.0 \
--port 8004 \
--tensor-parallel-size 16 \
--seed 1024 \
--enable-expert-parallel \
--max-num-seqs 8 \
--max-model-len 16384 \
--max-num-batched-tokens 2048 \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.95 \
--additional-config '{"ascend_scheduler_config":{"enabled":true, "enable_chunked_prefill": true}}' \
--enforce-eager

Output:

嗯，用户让我介绍一下联通公司。首先得确定用户需要哪些方面的信息。可能用户只是想了解基本概况，比如成立时间、主要业务，或者也可能对联通的市场地位、技术发展感兴趣。\n\n接下来，我需要回忆联通的基础信息。成立于1994年，原称中国联通，2009年和中国网通合并，改名为中国联通。这点要明确，避免混淆。然后总部在北京，央企属性很重要，说明其背景和稳定性。\n\n业务方面

✅ 2. AclGraph Mode (TP16+EP & Single Node)

Run:

vllm serve /root/.cache/modelscope/hub/models/ZhipuAI/GLM-4___5 \
--host 0.0.0.0 \
--port 8004 \
--tensor-parallel-size 16 \
--seed 1024 \
--enable-expert-parallel \
--max-num-seqs 8 \
--max-model-len 16384 \
--max-num-batched-tokens 2048 \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.95 \
--additional-config '{"ascend_scheduler_config":{"enabled":true, "enable_chunked_prefill": true}}'

Output:

嗯，用户让我介绍一下联通公司。首先得确定用户需要哪些方面的信息。可能用户只是想了解基本概况，比如成立时间、主要业务，或者也可能对联通的市场地位、技术发展感兴趣。\n\n接下来，我需要回忆联通的基础信息。成立于1994年，原称中国联通，2009年和中国网通合并，改名为中国联通。这点要明确，避免混淆。然后总部在北京，央企属性很重要，说明其背景和稳定性。\n\n业务方面

❌ 3. AclGraph Mode + Enable multistream_overlap_shared_expert (TP16+EP & Single Node)

Run:

vllm serve /root/.cache/modelscope/hub/models/ZhipuAI/GLM-4___5 \
--host 0.0.0.0 \
--port 8004 \
--tensor-parallel-size 16 \
--seed 1024 \
--enable-expert-parallel \
--max-num-seqs 8 \
--max-model-len 16384 \
--max-num-batched-tokens 2048 \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.95 \
--additional-config '{"ascend_scheduler_config":{"enabled":true, "enable_chunked_prefill": true}}' \
--additional-config '{"multistream_overlap_shared_expert": true}'

Output:

[rank7]:[E928 07:22:40.736525688 compiler_depend.ts:429] operator():build/CMakeFiles/torch_npu.dir/compiler_depend.ts:128 NPU function error: call aclnnMoeDistributeDispatchV2 failed, error code is 207001
[ERROR] 2025-09-28-07:22:30 (PID:316198, Device:7, RankID:-1) ERR00100 PTA call acl api failed
[Error]: Failed to apply for memory.
        Check the remaining storage space in the hardware environment.
EL0004: [PID: 316198] 2025-09-28-07:22:29.827.496 Failed to allocate memory.
        Possible Cause: Available memory is insufficient.
        Solution: Close applications not in use.
        TraceBack (most recent call last):
        Failed to allocate resource[DeviceMemory] with info [size:2147483648]. Reason: Memory resources are exhausted.
        Nnopbase fails to invoke the HcclAllocComResourceByTiling function of the hccl module. ret = 2, comm = 0xaaab1048cc60.
        Check nnopbase::NnopBaseHcclWrapper::GetInstance().HcclAllocComResourceByTiling(commHandle, stream, ((NnopbaseTilingData *)executor->args->tilingInfo.tilingData)->GetData(), &contextAddr) failed
        Check NnopbaseGetHcomResource(executor, stream) failed
        Check NnopbaseExecutorGetMc2Num(executor, stream, &argsAddr, &mc2Num) failed
        Check NnopbaseExecutorPrepareParamsExt(executor, stream) failed
        Check NnopbaseRunWithWorkspace(executor, stream, workspace, workspaceSize) failed
        Task execute failed, device_id=7, stream_id=112, task_id=0, flip_num=0, task_type=87.[FUNC:GetError][FILE:stream.cc][LINE:1183]
        Failed to synchronize stream, retCode=0x7150004.[FUNC:SyncGetDevMsg][FILE:api_impl.cc][LINE:6140]
        Sync get device msg failed, retCode=0x7150004.[FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6159]
        rtGetDevMsg execute failed, reason=[tsfw param illegal][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]

Exception raised from operator() at build/CMakeFiles/torch_npu.dir/compiler_depend.ts:128 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xd4 (0xffff800a3ea4 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xe4 (0xffff80043e44 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0x1582668 (0xfffe0df32668 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0x22887d4 (0xfffe0ec387d4 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x8fb170 (0xfffe0d2ab170 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x8fd504 (0xfffe0d2ad504 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x8f9e2c (0xfffe0d2a9e2c in /usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0xd31fc (0xffff7feb31fc in /lib/aarch64-linux-gnu/libstdc++.so.6)
frame #8: <unknown function> + 0x7d5b8 (0xffff8c17d5b8 in /lib/aarch64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0xe5edc (0xffff8c1e5edc in /lib/aarch64-linux-gnu/libc.so.6)

Fix PR: maybe #3163.

❌ 4. AclGraph Mode (TP8+DP2+EP & Multi Nodes)

Run (node0):

vllm serve /root/.cache/modelscope/hub/models/ZhipuAI/GLM-4___5 \
--host 0.0.0.0 \
--port 8004 \
--api-server-count 1 \
--data-parallel-size 2 \
--data-parallel-size-local 1 \
--data-parallel-address 172.22.0.155 \
--tensor-parallel-size 8 \
--data-parallel-rpc-port 13390 \
--seed 1024 \
--enable-expert-parallel \
--max-num-seqs 8 \
--max-model-len 16384 \
--max-num-batched-tokens 2048 \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.95 \
--additional-config '{"ascend_scheduler_config":{"enabled":true, "enable_chunked_prefill": true}}'

Run (node1):

vllm serve /root/.cache/modelscope/hub/models/ZhipuAI/GLM-4___5 \
--host 0.0.0.0 \
--port 8004 \
--api-server-count 1 \
--data-parallel-size 2 \
--data-parallel-size-local 1 \
--data-parallel-address 172.22.0.155 \
--tensor-parallel-size 8 \
--data-parallel-rpc-port 13390 \
--seed 1024 \
--enable-expert-parallel \
--max-num-seqs 8 \
--max-model-len 16384 \
--max-num-batched-tokens 2048 \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.95 \
--additional-config '{"ascend_scheduler_config":{"enabled":true, "enable_chunked_prefill": true}}' \
--headless \
--data-parallel-start-rank 1

Output:

(Worker_DP0_TP2_EP2 pid=342485) ERROR 09-28 08:08:41 [multiproc_executor.py:671] WorkerProc hit an exception.
(Worker_DP0_TP2_EP2 pid=342485) ERROR 09-28 08:08:41 [multiproc_executor.py:671] Traceback (most recent call last):
(Worker_DP0_TP2_EP2 pid=342485) ERROR 09-28 08:08:41 [multiproc_executor.py:671]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 666, in worker_busy_loop
(Worker_DP0_TP2_EP2 pid=342485) ERROR 09-28 08:08:41 [multiproc_executor.py:671]     output = func(*args, **kwargs)
(Worker_DP0_TP2_EP2 pid=342485) ERROR 09-28 08:08:41 [multiproc_executor.py:671]              ^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP2_EP2 pid=342485) ERROR 09-28 08:08:41 [multiproc_executor.py:671]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 194, in determine_available_memory
(Worker_DP0_TP2_EP2 pid=342485) ERROR 09-28 08:08:41 [multiproc_executor.py:671]     self.model_runner.profile_run()
(Worker_DP0_TP2_EP2 pid=342485) ERROR 09-28 08:08:41 [multiproc_executor.py:671]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2516, in profile_run
(Worker_DP0_TP2_EP2 pid=342485) ERROR 09-28 08:08:41 [multiproc_executor.py:671]     hidden_states = self._dummy_run(self.max_num_tokens,
(Worker_DP0_TP2_EP2 pid=342485) ERROR 09-28 08:08:41 [multiproc_executor.py:671]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP2_EP2 pid=342485) ERROR 09-28 08:08:41 [multiproc_executor.py:671]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(Worker_DP0_TP2_EP2 pid=342485) ERROR 09-28 08:08:41 [multiproc_executor.py:671]     return func(*args, **kwargs)
(Worker_DP0_TP2_EP2 pid=342485) ERROR 09-28 08:08:41 [multiproc_executor.py:671]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP2_EP2 pid=342485) ERROR 09-28 08:08:41 [multiproc_executor.py:671]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2352, in _dummy_run
(Worker_DP0_TP2_EP2 pid=342485) ERROR 09-28 08:08:41 [multiproc_executor.py:671]     _) = self._sync_metadata_across_dp(num_tokens, with_prefill, False)
(Worker_DP0_TP2_EP2 pid=342485) ERROR 09-28 08:08:41 [multiproc_executor.py:671]          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP2_EP2 pid=342485) ERROR 09-28 08:08:41 [multiproc_executor.py:671]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 826, in _sync_metadata_across_dp
(Worker_DP0_TP2_EP2 pid=342485) ERROR 09-28 08:08:41 [multiproc_executor.py:671]     max_tokens_across_dp = torch.max(num_tokens_across_dp).item()
(Worker_DP0_TP2_EP2 pid=342485) ERROR 09-28 08:08:41 [multiproc_executor.py:671]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP0_TP2_EP2 pid=342485) ERROR 09-28 08:08:41 [multiproc_executor.py:671] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is HcclAllreduce.

Fix PR: maybe #3208.

[wxsIcey]

✅ Qwen3-Next:

# isort: skip_file
import os

os.environ["VLLM_USE_MODELSCOPE"] = "True"
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"

from vllm import LLM, SamplingParams


def main():
    prompts = [
        "窗前明月光，",
        "The president of the United States is Mr.",
        "The capital of France is",
        "The future of AI is",
        "感时花溅泪，",
        "家书抵万金啥意思？",
        "plz tell me a story: ",
    ]

    # Create a sampling params object.
    sampling_params = SamplingParams(max_tokens=100, temperature=0.6, top_k=40, top_p=0.95)
    # Create an LLM.
    llm = LLM(
        model="Qwen/Qwen3-Next-80B-A3B-Instruct",
              tensor_parallel_size=4,
              enforce_eager=True,
              trust_remote_code=True,
              max_model_len=256,
              gpu_memory_utilization=0.7,
              block_size=64
              )

    # Generate texts from the prompts.
    outputs = llm.generate(prompts, sampling_params)
    for output in outputs:
        prompt = output.prompt
        generated_text = output.outputs[0].text
        print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")


if __name__ == "__main__":
    main()

Prompt: '窗前明月光，', Generated text: '疑是地上霜。\n举头望明月，低头思故乡。\n\n这首诗是唐代大诗人李白的《静夜思》。全诗如下：\n\n**《静夜思》**  \n唐·李白\n\n床前明月光，  \n疑是地上霜。  \n举头望明月，  \n低头思故乡。\n\n---\n\n### 诗句释义：\n\n- **“床前明月光”**：明亮的月光洒在地上。  \n  （“床”'
Prompt: 'The president of the United States is Mr.', Generated text: ' Joe Biden. His vice president is Kamala Harris, who is the first woman, first African-American, and first person of South Asian descent to hold this position. As the 46th president of the United States, Joe Biden was inaugurated as the 46th president of the United States on January 20, 2021, and Kamala Harris became the 49th vice president, making her the first female, first Black, and first South Asian American'
Prompt: 'The capital of France is', Generated text: ' Paris. What is the capital city of France?\n\nThe capital of France is Paris.'
Prompt: 'The future of AI is', Generated text: ' not just about building smarter machines—it’s about creating intelligent, adaptive, and ethically responsible systems that can collaborate with humans in meaningful, safe, and sustainable ways.\n\nTo achieve this, we need a new generation of AI: \n•\u202f\u202fHuman-Centered AI — Systems that understand context, empathy, and human intent.\n•\u202fRobust and Safe — Resilient, explainable, and aligned with human values.\n•\u2002Sustainable — Built with energy efficiency, fairness'
Prompt: '感时花溅泪，', Generated text: '恨别鸟惊心。  诗句的修辞和表达效果\n这句诗“感时花溅泪，惊鸟动心”（实际出自杜甫《春望》：“感时花溅泪，恨别鸟惊心。”）是唐代大诗人杜甫在《春望》一诗中的名句，其艺术表现力极强，蕴含深厚的情感与修辞技巧。\n\n我们来逐层分析：\n\n---\n\n🔹 **原句出处**（'
Prompt: '家书抵万金啥意思？', Generated text: '为什么叫家书？这个家字用的对吗？\n\n这是一个非常有趣、也很有文化深度的问题！\n\n你问得非常好：“**家书**”这个词，为什么用“家”？“家”和“家书”在古代汉语语境中，确实有非常深刻的文化内涵，我们来一一拆解。\n\n---\n\n### ✅ 一、先回答核心问题：\n\n### ❓ **“家书”中的“家”是什么意思？**\n👉 **不是'
Prompt: 'plz tell me a story: ', Generated text: '1000 words\n\n**Title: The Last Lighthouse Keeper of Gull’s Mire**\n\nThe sea did not forgive. Not after 1917. Not after the *Elderglow*.\n\nBut on the night the fog swallowed the Cliffs of Gullmire—when the moon hid and the sea breathed in sighs—Elias Wren, the last lighthouse keeper, stood at the edge of the cliff with a lantern in his hand and a letter'