-
Notifications
You must be signed in to change notification settings - Fork 565
Description
Your current environment
报错日志
WorkerProc hit an exception.^M
Traceback (most recent call last):^M
File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 666, in worker_busy_loop^M
output = func(*args, **kwargs)^M
^^^^^^^^^^^^^^^^^^^^^^M
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 381, in execute_dummy_batch^M
self.model_runner._dummy_run(^M
File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context^M
return func(*args, **kwargs)^M
^^^^^^^^^^^^^^^^^^^^^^M
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2494, in _dummy_run^M
hidden_states = self._generate_dummy_run_hidden_states(^M
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2325, in _generate_dummy_run_hidden_states^M
update_mla_attn_params(self.update_stream, forward_context,^M
File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/acl_graph.py", line 305, in update_mla_attn_params^M
torch.npu.graph_task_update_end(update_stream)^M
File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/npu/graphs.py", line 75, in graph_task_update_end^M
_graph_task_update_end(stream)^M
RuntimeError: graph_task_update_end:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:48 NPU function error: c10_npu::acl::AclmdlRICaptureTaskUpdateEnd(stream), error code is 107033^M
[ERROR] 2025-11-14-08:37:51 (PID:856555, Device:3, RankID:-1) ERR00100 PTA call acl api failed.^M
[PID: 856555] 2025-11-14-08:37:51.805.184 Invalid_Argument(EE1001): The argument is invalid.Reason: Unable to end update tasks because the stream is busy.^M
Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.^M
TraceBack (most recent call last):^M
rtMemcpy execute failed, reason=[the current capture mode does not support this operation][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]^M
synchronized memcpy failed, kind = 1, runtime result = 107030[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162]^M
The argument is invalid.Reason: Unable to end update tasks because the stream is busy.^M
rtsStreamEndTaskUpdate execute failed, reason=[task group status error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]^M
end update capture task group failed, runtime result = 107033[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162]^M
🐛 Describe the bug
vllm:0.11.0rc1
cann:8.3.rc1
PTA:2.7.1
HDK:25.3.rc1