Skip to content

[Bug]: v0.11.0rc1部署910c单机部署deepseekv3.1 w8a8同时开启mtp和整图推理报错 #4208

@ivan-zouming

Description

@ivan-zouming

Your current environment

报错日志
WorkerProc hit an exception.^M
Traceback (most recent call last):^M
File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 666, in worker_busy_loop^M
output = func(*args, **kwargs)^M
^^^^^^^^^^^^^^^^^^^^^^M
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 381, in execute_dummy_batch^M
self.model_runner._dummy_run(^M
File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context^M
return func(*args, **kwargs)^M
^^^^^^^^^^^^^^^^^^^^^^M
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2494, in _dummy_run^M
hidden_states = self._generate_dummy_run_hidden_states(^M
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2325, in _generate_dummy_run_hidden_states^M
update_mla_attn_params(self.update_stream, forward_context,^M
File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/acl_graph.py", line 305, in update_mla_attn_params^M
torch.npu.graph_task_update_end(update_stream)^M
File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/npu/graphs.py", line 75, in graph_task_update_end^M
_graph_task_update_end(stream)^M
RuntimeError: graph_task_update_end:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:48 NPU function error: c10_npu::acl::AclmdlRICaptureTaskUpdateEnd(stream), error code is 107033^M
[ERROR] 2025-11-14-08:37:51 (PID:856555, Device:3, RankID:-1) ERR00100 PTA call acl api failed.^M
[PID: 856555] 2025-11-14-08:37:51.805.184 Invalid_Argument(EE1001): The argument is invalid.Reason: Unable to end update tasks because the stream is busy.^M
Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.^M
TraceBack (most recent call last):^M
rtMemcpy execute failed, reason=[the current capture mode does not support this operation][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]^M
synchronized memcpy failed, kind = 1, runtime result = 107030[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162]^M
The argument is invalid.Reason: Unable to end update tasks because the stream is busy.^M
rtsStreamEndTaskUpdate execute failed, reason=[task group status error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]^M
end update capture task group failed, runtime result = 107033[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162]^M

🐛 Describe the bug

vllm:0.11.0rc1
cann:8.3.rc1
PTA:2.7.1
HDK:25.3.rc1

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions