[Bug]: `copy_between_host_and_device_opapi` error when enable `VLLM_ASCEND_ENABLE_FLASHCOMM1=1`

[shen-shanshan]

Your current environment

The output of `python collect_env.py`

vllm: v0.11.0
vllm-ascend: v0.11.0-dev

🐛 Describe the bug

Run:

export VLLM_ASCEND_ENABLE_FLASHCOMM1=1

vllm serve /root/.cache/modelscope/hub/models/Qwen/Qwen2.5-VL-7B-Instruct \
--max_model_len 16384 \
--max-num-batched-tokens 16384 \
--tensor-parallel-size 4

It directly broke:

(Worker_TP0 pid=51886) ERROR 11-14 08:13:55 [multiproc_executor.py:671] WorkerProc hit an exception.
(Worker_TP0 pid=51886) ERROR 11-14 08:13:55 [multiproc_executor.py:671] Traceback (most recent call last):
(Worker_TP0 pid=51886) ERROR 11-14 08:13:55 [multiproc_executor.py:671]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 666, in worker_busy_loop
(Worker_TP0 pid=51886) ERROR 11-14 08:13:55 [multiproc_executor.py:671]     output = func(*args, **kwargs)
(Worker_TP0 pid=51886) ERROR 11-14 08:13:55 [multiproc_executor.py:671]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 227, in determine_available_memory
(Worker_TP0 pid=51886) ERROR 11-14 08:13:55 [multiproc_executor.py:671]     self.model_runner.profile_run()
(Worker_TP0 pid=51886) ERROR 11-14 08:13:55 [multiproc_executor.py:671]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2557, in profile_run
(Worker_TP0 pid=51886) ERROR 11-14 08:13:55 [multiproc_executor.py:671]     hidden_states = hidden_states[logit_indices]
(Worker_TP0 pid=51886) ERROR 11-14 08:13:55 [multiproc_executor.py:671] RuntimeError: copy_between_host_and_device_opapi:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:56 NPU function error: SUSPECT REMOTE ERROR, error code is 507057

[shen-shanshan]

I also get another error when running:

export PYTORCH_NPU_ALLOC_CONF="max_split_size_mb:250"
export CPU_AFFINITY_CONF=2
export MALLOC_CONF='background_thread:true,metadata_thp:auto,dirty_decay_ms:30000,muzzy_decay_ms:30000,abort_conf:true,percpu_arena:phycpu'
export VLLM_RPC_TIMEOUT=120
export TASK_QUEUE_ENABLE=1
export HCCL_OP_EXPANSION_MODE=AIV
export VLLM_ASCEND_ENABLE_FLASHCOMM=1
export VLLM_ASCEND_ENABLE_PREFETCH_MLP=1
export VLLM_ASCEND_MLP_GATE_UP_PREFETCH_SIZE=$((18*1024*1024))
export VLLM_ASCEND_MLP_DOWN_PREFETCH_SIZE=$((18*1024*1024))
export VLLM_LOGGING_LEVEL=WARNING
export USE_OPTIMIZED_MODEL=1
export ASCEND_LAUNCH_BLOCKING=1

vllm serve /root/.cache/modelscope/hub/models/Qwen/Qwen2.5-VL-7B-Instruct \
--max_model_len 16384 \
--max-num-batched-tokens 16384 \
--tensor-parallel-size 4 \
--enforce-eager \
--max-num-seqs 16 \
--dtype bfloat16 \
--trust-remote-code \
--gpu-memory-utilization 0.9

log:

(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671] WorkerProc hit an exception.
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671] Traceback (most recent call last):
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 666, in worker_busy_loop
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     output = func(*args, **kwargs)
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 227, in determine_available_memory
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     self.model_runner.profile_run()
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2530, in profile_run
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     hidden_states = self._dummy_run(self.max_num_tokens,
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     return func(*args, **kwargs)
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2493, in _dummy_run
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     hidden_states = self._generate_dummy_run_hidden_states(
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2314, in _generate_dummy_run_hidden_states
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     hidden_states = self.model(input_ids=input_ids,
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2_5_vl.py", line 1450, in forward
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     hidden_states = self.language_model.model(
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/vllm-workspace/vllm/vllm/compilation/decorators.py", line 225, in __call__
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     return self.forward(*args, **kwargs)
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 364, in forward
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     hidden_states, residual = layer(positions, hidden_states, residual)
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 257, in forward
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     hidden_states = self.self_attn(
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen2.py", line 186, in forward
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     q, k = self.rotary_emb(positions, q, k)
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/vllm-workspace/vllm/vllm/model_executor/custom_op.py", line 44, in forward
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     return self._forward_method(*args, **kwargs)
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/vllm-workspace/vllm-ascend/vllm_ascend/ops/rotary_embedding.py", line 423, in forward_oot
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     query, key = torch_npu.npu_mrope(positions,
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]   File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/_ops.py", line 1158, in __call__
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671]     return self._op(*args, **(kwargs or {}))
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671] RuntimeError: RunOpApiV2:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:251 NPU function error: SUSPECT REMOTE ERROR, error code is 507057
(Worker_TP0 pid=68602) ERROR 11-14 09:16:55 [multiproc_executor.py:671] [ERROR] 2025-11-14-09:16:55 (PID:68602, Device:0, RankID:-1) ERR00100 PTA call acl api failed.