[v0.13.0rc2] FAQ / Feedback | 问题/反馈

[wangxiyuan]

Anything you want to discuss about vllm on ascend.

Please use doc: https://docs.vllm.ai/projects/ascend/en/v0.13.0/

请使用 https://docs.vllm.ai/projects/ascend/en/v0.13.0/安装

[zhangxinyuehfad]

When using the docker image from quay.io, you should rebuild vllm-ascend(v0.13.0) downgrade torch-npu to 2.8.0 before starting serve

pip install torch-npu==2.8.0

According to this document, the error is reported:
Qwen3-VL-30B-A3B-Instruct model also has the same problem

vllm serve Qwen/Qwen3-30B-A3B --tensor-parallel-size 2 --enable_expert_parallel

(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824] WorkerProc hit an exception.
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824] Traceback (most recent call last):
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]   File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/acl_graph.py", line 155, in __call__
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]     output = self.runnable(*args, **kwargs)
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]   File "/vllm-workspace/vllm/vllm/compilation/piecewise_backend.py", line 178, in __call__
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]     return range_entry.runnable(*args)
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]     return fn(*args, **kwargs)
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 1241, in forward
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]     return compiled_fn(full_args)
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 384, in runtime_wrapper
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]     all_outs = call_func_at_runtime_with_args(
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]     out = normalize_as_list(f(args))
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]                             ^^^^^^^
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 750, in inner_fn
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]     outs = compiled_fn(args)
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 556, in wrapper
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]     return compiled_fn(runtime_args)
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/utils.py", line 100, in g
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]     return f(*args)
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]            ^^^^^^^^
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/graph_module.py", line 848, in call_wrapped
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]     return self._wrapped_call(self, *args, **kwargs)
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/graph_module.py", line 424, in __call__
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]     raise e
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/graph_module.py", line 411, in __call__
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]     return self._call_impl(*args, **kwargs)
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]     return forward_call(*args, **kwargs)
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]   File "<eval_with_key>.155 from /usr/local/python3.11.14/lib/python3.11/site-packages/torch/fx/experimental/proxy_tensor.py:1301 in wrapped", line 28, in forward
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]     qkv_rmsnorm_rope_default = torch.ops.vllm.qkv_rmsnorm_rope.default(mm_2, arg11_1, arg10_1, arg8_1, arg9_1, 1024, 128, 128, 1e-06);  mm_2 = arg11_1 = arg10_1 = arg8_1 = arg9_1 = None
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 829, in __call__
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]     return self._op(*args, **kwargs)
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]   File "/vllm-workspace/vllm-ascend/vllm_ascend/ops/triton/linearnorm/split_qkv_rmsnorm_rope.py", line 239, in split_qkv_rmsnorm_rope_impl
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]     split_qkv_rmsnorm_rope_kernel[(n_rows, n_cols, 1)](
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/triton/runtime/jit.py", line 353, in <lambda>
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]     return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/triton/runtime/jit.py", line 592, in run
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]     stream = driver.active.get_current_stream(device)
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/triton/backends/ascend/driver.py", line 219, in get_current_stream
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]     return _npu_getCurrentRawStream(device)
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is HcclAllreduce.
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
(Worker_TP0_EP0 pid=75832) ERROR 01-26 07:59:55 [multiproc_executor.py:824] [ERROR] 2026-01-26-07:59:54 (PID:75832, Device:0, RankID:-1) ERR00100 PTA call acl api failed.

But the following command runs ok:

vllm serve /root/.cache/modelscope/hub/models/Qwen/Qwen3-30B-A3B \
--max_model_len 16384 \
--max-num-batched-tokens 16384 \
--max-num-seqs 8 \
--tensor-parallel-size 2 \
--enable-expert-parallel

root@liteserver-for-vllm-ascend-00001:/workspace# curl -X POST "http://0.0.0.0:8000/v1/chat/completions" -H "Content-Type: application/json" -d '{
  "messages": [
    {"role": "user", "content": "The future of AI is"}
  ]
}'
{"id":"chatcmpl-b16a2187e17733a1","object":"chat.completion","created":1769415051,"model":"/root/.cache/modelscope/hub/models/Qwen/Qwen3-30B-A3B","choices":[{"index":0,"message":{"role":"assistant","content":"<think>\nOkay, the user asked \"The future of AI is,\" and I need to respond. First, I should figure out what they're really looking for. Maybe they want an overview of where AI is heading, but the question is incomplete. They might be expecting a list of possibilities or predictions.\n\nI should consider different aspects of AI's future. Let me think about the key areas. There's technological advancements, like more powerful models, maybe general AI. Then there are applications in various fields—healthcare, education, industry. Also, ethical issues, job displacement, regulation. Maybe societal impacts, like changes in daily life or global economy.\n\nWait, the user might not have a specific angle. I should cover the main points without being too technical. Also, need to mention both opportunities and challenges. Maybe start with the positive developments, then address the concerns. But how to structure this?\n\nI should check if there are any recent trends or breakthroughs that are significant. Like the rise of large language models, AI in climate change solutions, or ethical frameworks being developed. Also, the balance between innovation and regulation is important.\n\nAvoid making it too long, but comprehensive. Maybe use bullet points or sections. But since the user might want a continuous text, I'll structure it with clear paragraphs. Also, make sure to mention that the future is uncertain and depends on various factors.\n\nWait, the user might be a student, researcher, or just someone curious. Need to keep it accessible. Avoid jargon. Highlight both the potential benefits and the risks. Maybe end with a balanced view, emphasizing responsible development.\n\nI should also mention specific examples, like AI in healthcare diagnostics, autonomous vehicles, personalized education. But also touch on issues like bias, privacy, job automation. Maybe include the role of international cooperation in AI governance.\n\nDouble-check for any recent developments post-2023 that might be relevant. For example, advancements in AI ethics, new regulations, or major projects. But since my knowledge is up to 2023, I can't include 2024 data. So stick to what's known up to that point.\n\nAlso, consider the user's possible deeper needs. They might be looking for reassurance, information for a project, or just general knowledge. The response should be informative but not overwhelming. Make sure to highlight the transformative potential while acknowledging the challenges.\n</think>\n\nThe future of AI is a dynamic and multifaceted landscape, shaped by rapid technological advancements, ethical considerations, and societal needs. Here’s a structured overview of key trends and possibilities:\n\n### **1. Technological Advancements**  \n- **General AI (AGI):** While still speculative, research into more versatile, human-like AI systems continues. Breakthroughs in neural networks, quantum computing, and neuroscience could accelerate progress.  \n- **Specialized AI:** AI will become increasingly specialized, excelling in domains like climate modeling, drug discovery, and personalized education.  \n- **Human-AI Collaboration:** Tools like AI-assisted creativity (e.g., art, music) and decision-making systems will enhance human capabilities rather than replace them.  \n\n### **2. Societal and Economic Impacts**  \n- **Workforce Transformation:** AI will automate routine tasks, reshaping industries and creating new roles. Upskilling and education will be critical to adapt to this shift.  \n- **Healthcare Revolution:** AI-driven diagnostics, personalized treatments, and robotic surgery could significantly improve patient outcomes and accessibility.  \n- **Smart Infrastructure:** Cities will leverage AI for traffic management, energy efficiency, and public safety, fostering sustainable urban development.  \n\n### **3. Ethical and Regulatory Challenges**  \n- **Bias and Fairness:** Addressing algorithmic bias and ensuring transparency will be essential to prevent discrimination and build trust.  \n- **Privacy Concerns:** AI’s reliance on data raises questions about surveillance, consent, and data security, necessitating robust frameworks.  \n- **Global Governance:** International cooperation will be vital to establish standards for AI safety, accountability, and equitable access.  \n\n### **4. Risks and Opportunities**  \n- **Existential Risks:** Long-term concerns about superintelligent AI (e.g., misalignment with human values) remain, though many experts emphasize immediate, practical risks.  \n- **Environmental Impact:** AI’s energy consumption and carbon footprint will require innovations in sustainable computing.  \n- **Positive Potential:** AI could tackle global challenges like climate change, food security, and pandemic response, fostering a more resilient world.  \n\n### **5. Cultural and Philosophical Shifts**  \n- **Redefining Creativity:** AI’s role in art, literature, and culture may challenge traditional notions of authorship and originality.  \n- **Human Identity:** As AI becomes more integrated into daily life, questions about autonomy, agency, and the essence of \"human\" will persist.  \n\n### **Conclusion**  \nThe future of AI is not predetermined but shaped by choices made today. Balancing innovation with ethics, inclusivity, and sustainability will determine whether AI becomes a force for collective good or a source of division. Collaboration among governments, researchers, and communities will be key to navigating this transformative era.  \n\nWhat aspects of AI’s future interest you most?","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning":null,"reasoning_content":null},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":13,"total_tokens":1080,"completion_tokens":1067,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}root@liteserver-for-vllm-ascend-00001:/workspace#

[zhangxinyuehfad]

you should rebuild vllm-ascend(v0.13.0) downgrade torch-npu to 2.8.0 before starting serve

According to this document, the error is reported: Qwen3-VL-30B-A3B-Instruct model also has the same problem

vllm serve Qwen/Qwen3-30B-A3B --tensor-parallel-size 2 --enable_expert_parallel

This command needs to add --compilation-config '{"cudagraph_capture_sizes":[1,2,4,8]}

[Yikun]

This is the final release of v0.13.0 for vLLM Ascend. Please follow the official doc to get started.

Highlights

Model Support

• Qwen3-Next: Full support for Qwen3-Next series including 80B-A3B-Instruct with full graph mode, MTP, quantization (W8A8), NZ optimization, and chunked prefill. Fixed multiple accuracy and stability issues. #3450 #3572 #3428 #3918 #4058 #4245 #4070 #4477 #4770
• DeepSeek-R1 & DeepSeek-V3.2: Added support for DeepSeek-R1 multimodal capabilities and improved DeepSeek-V3.2 with MTP support, performance optimizations, and async scheduling enhancements. #3631 #3900 #3908 #4191 #4805
• InternVL: Added support for InternVL models with comprehensive e2e tests and accuracy evaluation. #3796 #3964
• LongCat-Flash: Added support for LongCat-Flash model. #3833
• minimax_m2: Added support for minimax_m2 model. #5624
• Whisper & Cross-Attention: Added support for cross-attention and Whisper models. #5592
• Pooling Models: Added support for pooling models with PCP adaptation and fixed multiple pooling-related bugs. #3122 #4143 #6056 #6057 #6146
• PanguUltraMoE: Added support for PanguUltraMoE model. #4615

Core Features

• Context Parallel (PCP/DCP): [Experimental] Added comprehensive support for Prefill Context Parallel (PCP) and Decode Context Parallel (DCP) with ACLGraph, MTP, chunked prefill, MLAPO, and Mooncake connector integration. This is an experimental feature - feedback welcome. #3260 #3731 #3801 #3980 #4066 #4098 #4183 #5672
• Full Graph Mode (ACLGraph): Enhanced full graph mode with GQA support, memory optimizations, unified logic between ACLGraph and Torchair, and improved stability. #3560 #3970 #3812 #3879 #3888 #3894 #5118
• Multi-Token Prediction (MTP): Significantly improved MTP support with chunked prefill for DeepSeek, quantization support, full graph mode, PCP/DCP integration, and async scheduling. MTP now works in most cases and is recommended for use. #2711 #2713 #3620 #3845 #3910 #3915 #4102 #4111 #4770 #5477
• Eagle Speculative Decoding: Eagle spec decode now works with full graph mode and is more stable. #5118 #4893 #5804
• PD Disaggregation: Set ADXL engine as default backend for disaggregated prefill with improved performance and stability. Added support for KV NZ feature for DeepSeek decode node. #3761 #3950 #5008 #3072
• KV Pool & Mooncake: Enhanced KV pool with Mooncake connector support for PCP/DCP, multiple input suffixes, and improved performance of Layerwise Connector. #3690 #3752 #3849 #4183 #5303
• EPLB (Elastic Prefill Load Balancing): EPLB is now more stable with many bug fixes. Mix placement now works. #6086
• Full Decode Only Mode: Added support for Qwen3-Next and DeepSeekv32 in full_decode_only mode with bug fixes. #3949 #3986 #3763
• Model Runner V2: Added basic support for Model Runner V2, the next generation of vLLM. It will be used by default in future releases. #5210

Features

• W8A16 Quantization: Added new W8A16 quantization method support. #4541
• UCM Connector: Added UCMConnector for KV Cache Offloading. #4411
• Batch Invariant: Implemented basic framework for batch invariant feature. #5517
• Sampling: Enhanced sampling with async_scheduler and disable_padded_drafter_batch support in Eagle. #4893

Hardware and Operator Support

• Custom Operators: Added multiple custom operators including:
  • Fused matmul/reduce-scatter kernel #3693
  • mrope fusion op #3708
  • Triton chunk_gated_delta_rule ops for Qwen3-Next #4070
  • l2norm triton kernel #4595
  • RejectSampler, MoeInitRoutingCustom, DispatchFFNCombine custom ops
• Operator Fusion: Added AddRmsnormQuant fusion pattern with SP support and inductor fusion for quantization. #5077 #4168
• MLA/SFA: Refactored SFA into MLA architecture for better maintainability. #3769
• FIA Operator: Adapted to npu_fused_infer_attention_score with flash decoding function. To optimize performance in small batch size scenarios, this attention operator is now available. Please refer to item 22 in FAQs to enable it. #4025
• CANN 8.5 Support: Removed CP redundant variables after FIA operator enables for CANN 8.5. #6039

Performance

Many custom ops and triton kernels were added in this release to speed up model performance:

• DeepSeek Performance: Improved performance for DeepSeek V3.2 by eliminating HD synchronization in async scheduling and optimizing memory usage for MTP. #4805 #2713
• Qwen3-Next Performance: Improved performance with Triton ops and optimizations. #5664 #5984 #5765
• FlashComm: Enhanced FlashComm v2 optimization with o_shared linear and communication domain fixes. #3232 #4188 #4458 #5848
• MoE Optimization: Optimized all2allv for MoE models and enhanced all-reduce skipping logic. #3738 #5329
• Attention Optimization: Moved attention update stream out of loop, converted BSND to TND format for long sequence optimization, and removed transpose step after attention switching to transpose_batchmatmul. #3848 #3778 #5390
• Quantization Performance: Moved quantization before allgather in Allgather EP. #3420
• Layerwise Connector: Improved performance of Layerwise Connector. #5303
• Prefix Cache: Improved performance of prefix cache features. #4022
• Async Scheduling: Fixed async copy and eliminated hangs in async scheduling. #4113 #4233
• Memory Operations: Removed redundant D2H operations and deleted redundant operations in model_runner. #4063 #3677
• Rope Embedding: Optimized rope embedding with triton kernel for huge performance gain. #5918
• Sampling: Added support for advanced apply_top_k_top_p without top_k constraint. #6098
• Multimodal: Parallelized Q/K/V padding in AscendMMEncoderAttention for better performance. #6204

Dependencies

• CANN: Upgraded to 8.5.0 #6112
• torch-npu: Upgraded to 2.8.0.post1. Please note that the post version will not be installed by default. Please install it by hand from pypi mirror. #3896 #4433
• triton-ascend: Upgraded to 3.2.0 #6105
• vLLM: Upgraded to 0.13.0 and dropped 0.12.0 support. #5146
• Transformers: Upgraded to >= 4.57.3 #5250

Deprecation & Breaking Changes

• CPUOffloadingConnector is deprecated. We'll remove it in the next release. It'll be replaced by CPUOffload feature from vLLM in the future.
• EPLB config options have been moved to eplb_config in additional config. The old ones will be removed in the next release.
• ProfileExecuteDuration feature is deprecated. It's replaced by ObservabilityConfig from vLLM.
• Ascend Scheduler has been dropped. #4623
• Torchair has been dropped. #4814
• VLLM_ASCEND_ENABLE_DENSE_OPTIMIZE is removed and VLLM_ASCEND_ENABLE_PREFETCH_MLP is recommended to replace as they were always enabled together. #5272
• VLLM_ENABLE_FUSED_EXPERTS_ALLGATHER_EP is dropped now. #5270
• VLLM_ASCEND_ENABLE_NZ is disabled for float weight case, since we noticed that the performance is not good in some float cases. Feel free to set it to 2 if you make sure it works for your case. #4878
• chunked_prefill_for_mla in additional_config is dropped now. #5296
• dump_config in additional_config is renamed to dump_config_path and the type is changed from dict to string. #5296
• --task parameter for embedding models is deprecated. #5257
• The value of VLLM_ASCEND_ENABLE_MLAPO env will be set to True by default in the next release. It'll be enabled in decode node by default. Please note that this feature will cost more memory. If you are memory sensitive, please set it to False.

Documentation

• Added comprehensive developer guides for ACLGraph, MTP, KV Pool, EPLB, and PD disaggregation features
• Added tutorials for multiple models including DeepSeek-V3.2-Exp, Qwen3-Next, and various multimodal models
• Updated FAQ and configuration documentation

Others

• OOM Fix: OOM error on VL models is fixed now. We're keeping observing it. If you hit OOM problem again, please submit an issue. #5136
• Qwen3-Next-MTP Accuracy: Fixed an accuracy bug of Qwen3-Next-MTP when batched inferring. #4932
• ZMQ Bug Fix: Fixed zmq send/receive failed bug. #5503
• Weight Transpose: Fixed weight transpose in RL scenarios. #5567
• Eagle3 SP: Adapted SP to eagle3. #5562
• GLM4.6 MTP: GLM4.6 now supports MTP with fullgraph. #5460
• Flashcomm2 Oshard: Flashcomm2 now works with oshard generalized feature. #4723
• Fine-grained Shared Expert Overlap: Support fine-grained shared expert overlap. #5962

Known Issues

• Qwen3-Next doesn't support long sequence scenario, and we should limit gpu-memory-utilization according to the doc to run Qwen3-Next. We'll improve it in the next release.
• The functional break on Qwen3-Next when the input/output is around 3.5k/1.5k is fixed, but it introduces a regression on performance. We'll fix it in the next release. #5357
• There is a precision issue with curl on ultra-short sequences in DeepSeek-V3.2. We'll fix it in the next release. #5370