[Bug]: DeepSeek-V3.2-Exp-w8a8性能测试torch_npu.npu_scatter_nd_update_算子报错 #4296
Description
Your current environment
CANN:
package_name=Ascend-cann-toolkit
version=8.3.RC1
innerversion=V100R001C23SPC001B235
compatible_version=[V100R001C15],[V100R001C18],[V100R001C19],[V100R001C20],[V100R001C21],[V100R001C23]
arch=x86_64
os=linux
path=/usr/local/Ascend/ascend-toolkit/8.3.RC1/x86_64-linux
torch==2.7.1+cpu
torch_npu==2.7.1
torchvision==0.22.1+cpu
transformers==4.57.1
vLLM Version: 0.11.0
vLLM Ascend Version: 0.11.0rc1
NPU:910B2C 16卡 64G
🐛 Describe the bug
DeepSeek-V3.2-Exp-w8a8性能测试(4并发输入输出为2048)torch_npu.npu_scatter_nd_update_算子报错
VLLM服务启动脚本：
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
export model_path=/data/models/DeepSeek-V3.2-Exp-w8a8
export npu_count=16
export model_name=deepseek_v3.2
export port=8000
export ASCEND_LAUNCH_BLOCKING=1
vllm serve --port $port $model_path
--tensor-parallel-size $npu_count
--seed 1024
--quantization ascend
--served-model-name $model_name
--max-num-seqs 16
--max-model-len 13000
--max-num-batched-tokens 13000
--trust-remote-code
--no-enable-prefix-caching
--gpu-memory-utilization 0.8
--enforce-eager
报错日志信息如下：
(APIServer pid=221178) INFO: 127.0.0.1:34822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=221178) INFO: 127.0.0.1:34824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] WorkerProc hit an exception.
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] Traceback (most recent call last):
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/vllm/v1/executor/multiproc_executor.py", line 666, in worker_busy_loop
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] output = func(*args, **kwargs)
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/vllm_ascend/worker/worker_v1.py", line 276, in execute_model
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] output = self.model_runner.execute_model(scheduler_output,
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] return func(*args, **kwargs)
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/vllm_ascend/worker/model_runner_v1.py", line 1947, in execute_model
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] hidden_states = self._generate_process_reqs_hidden_states(
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/vllm_ascend/worker/model_runner_v1.py", line 1550, in _generate_process_reqs_hidden_states
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] hidden_states = self.model(
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] return self._call_impl(*args, **kwargs)
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] return forward_call(*args, **kwargs)
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/vllm/model_executor/models/deepseek_v2.py", line 1295, in forward
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] hidden_states = self.model(input_ids, positions, intermediate_tensors,
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/vllm/compilation/decorators.py", line 225, in call
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] return self.forward(*args, **kwargs)
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/vllm/model_executor/models/deepseek_v2.py", line 1171, in forward
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] hidden_states, residual = layer(positions, hidden_states, residual)
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] return self._call_impl(*args, **kwargs)
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] return forward_call(*args, **kwargs)
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/vllm/model_executor/models/deepseek_v2.py", line 1070, in forward
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] hidden_states = self.self_attn(
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] return self._call_impl(*args, **kwargs)
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] return forward_call(*args, **kwargs)
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/vllm_ascend/models/deepseek_v3_2.py", line 387, in forward
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] return self.sfa_attn(positions, hidden_states, kv_cache, attn_metadata)
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] return self._call_impl(*args, **kwargs)
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] return forward_call(*args, **kwargs)
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/vllm_ascend/models/layers/sfa.py", line 143, in forward
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] torch.ops.vllm.sfa_forward(hidden_states, need_gather_q_kv, output,
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/_ops.py", line 1158, in call
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] return self._op(*args, **(kwargs or {}))
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/vllm_ascend/models/layers/sfa.py", line 162, in sfa_forward
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] self.sfa_attn.impl.forward(hidden_states, kv_cache, attn_metadata,
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/vllm_ascend/attention/sfa_v1.py", line 826, in forward
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] decode_preprocess_res, prefill_preprocess_res = self._sfa_preprocess(
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/vllm_ascend/attention/sfa_v1.py", line 690, in sfa_preprocess
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] topk_indices = self.indexer_select(hidden_states_decode,
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/vllm_ascend/attention/sfa_v1.py", line 969, in indexer_select
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] torch_npu.npu_scatter_nd_update(kv_cache[2].view(-1, k.shape[-1]),
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/_ops.py", line 1158, in call
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] return self._op(*args, **(kwargs or {}))
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] RuntimeError: RunOpApiV2:torch_npu/csrc/framework/OpCommand.cpp:251 NPU function error: c10_npu::acl::AclrtSynchronizeStreamWithTimeout(stream), error code is 507035
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] [ERROR] 2025-11-20-02:28:06 (PID:221636, Device:3, RankID:-1) ERR00100 PTA call acl api failed
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] [Error]: The vector core execution is abnormal.
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] Rectify the fault based on the error information in the ascend log.
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] EZ9999: Inner Error!
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] EZ9999[PID: 221636] 2025-11-20-02:28:06.889.550 (EZ9999): The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 4, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x7006fb01df, mte error info: 0xa7030468f0, ifu error info: 0x9fd9b8cc0e00, ccu error info: 0xa77273aa10870038, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] TraceBack (most recent call last):
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30468f0, fixp_error1 info: 0xa7, fsmId:0, tslot:0, thread:0, ctxid:0, blk:21, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 7, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x91ec470ac, mte error info: 0xa70304b8f0, ifu error info: 0x3101b4b7c4c0, ccu error info: 0xe469f8394ad97f9c, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x304b8f0, fixp_error1 info: 0xa7, fsmId:0, tslot:0, thread:0, ctxid:0, blk:24, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 9, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0xd100f0c7c8, mte error info: 0xa7030468f0, ifu error info: 0x2b873122c49c0, ccu error info: 0xe1ee018c00500935, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30468f0, fixp_error1 info: 0xa7, fsmId:0, tslot:0, thread:0, ctxid:0, blk:26, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 10, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0xf111154249, mte error info: 0xa70305b8f0, ifu error info: 0x3289c91001c00, ccu error info: 0x3c3d2a420a868a76, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x305b8f0, fixp_error1 info: 0xa7, fsmId:0, tslot:0, thread:0, ctxid:0, blk:27, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 19, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x621416a16b, mte error info: 0xa7030558f0, ifu error info: 0x62d2b76e3fc40, ccu error info: 0x2cc5a1ec7ee6ebbf, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30558f0, fixp_error1 info: 0xa7, fsmId:0, tslot:0, thread:0, ctxid:0, blk:36, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 20, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0xe310553267, mte error info: 0xa70305e8f0, ifu error info: 0x66d91340f4780, ccu error info: 0xb049c20d09078018, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x305e8f0, fixp_error1 info: 0xa7, fsmId:0, tslot:0, thread:0, ctxid:0, blk:37, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 21, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x73175cd348, mte error info: 0xa70304c8f0, ifu error info: 0x7dd1d798fff00, ccu error info: 0x9017c3cc49c120d1, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x304c8f0, fixp_error1 info: 0xa7, fsmId:0, tslot:0, thread:0, ctxid:0, blk:38, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 23, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x7b0ed130e5, mte error info: 0xa70304b8f0, ifu error info: 0x463d31baeee40, ccu error info: 0x250c1b6107d28531, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x304b8f0, fixp_error1 info: 0xa7, fsmId:0, tslot:0, thread:0, ctxid:0, blk:40, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 27, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0xad0736ca6d, mte error info: 0xa70304c8f0, ifu error info: 0x1711827e68040, ccu error info: 0xd968046212a3054, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x304c8f0, fixp_error1 info: 0xa7, fsmId:0, tslot:0, thread:0, ctxid:0, blk:44, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 28, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x141c968410, mte error info: 0xa7030648f0, ifu error info: 0x19fb99818fa80, ccu error info: 0x84200a1c400f0d8c, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30648f0, fixp_error1 info: 0xa7, fsmId:0, tslot:0, thread:0, ctxid:0, blk:45, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 29, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x7a01784b0b, mte error info: 0xa7030498f0, ifu error info: 0x119218c290980, ccu error info: 0x913943a43ec0bb98, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30498f0, fixp_error1 info: 0xa7, fsmId:0, tslot:0, thread:0, ctxid:0, blk:46, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 31, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x790714c280, mte error info: 0xa7030498f0, ifu error info: 0x34001812b3cc0, ccu error info: 0x38fc3ace38053989, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30498f0, fixp_error1 info: 0xa7, fsmId:0, tslot:0, thread:0, ctxid:0, blk:0, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 34, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x6101318a67, mte error info: 0xa7030778f0, ifu error info: 0x40bd7920bc00, ccu error info: 0xe05df885046274cf, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30778f0, fixp_error1 info: 0xa7, fsmId:0, tslot:0, thread:0, ctxid:0, blk:1, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 39, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x721eb01a28, mte error info: 0xa7030468f0, ifu error info: 0x6438db01180c0, ccu error info: 0xc2f000215e28833b, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30468f0, fixp_error1 info: 0xa7, fsmId:0, tslot:0, thread:0, ctxid:0, blk:6, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 42, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x651c83e814, mte error info: 0xa70304b8f0, ifu error info: 0x2f5ac7bc2b40, ccu error info: 0x7e0c18c42b009e03, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x304b8f0, fixp_error1 info: 0xa7, fsmId:0, tslot:0, thread:0, ctxid:0, blk:9, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 43, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x6403912820, mte error info: 0xa70305e8f0, ifu error info: 0xf00086900800, ccu error info: 0xd16388af73c6481c, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x305e8f0, fixp_error1 info: 0xa7, fsmId:0, tslot:0, thread:0, ctxid:0, blk:10, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 44, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x6008820398, mte error info: 0xa70304a8f0, ifu error info: 0x71dfb0d680180, ccu error info: 0x66a4485819c10139, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x304a8f0, fixp_error1 info: 0xa7, fsmId:0, tslot:0, thread:0, ctxid:0, blk:11, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 45, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x4109d4445c, mte error info: 0xa70307d8f0, ifu error info: 0xcf9609e9a080, ccu error info: 0xd8000647c06844c, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x307d8f0, fixp_error1 info: 0xa7, fsmId:0, tslot:0, thread:0, ctxid:0, blk:12, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 47, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0xe113a78a94, mte error info: 0xa7030468f0, ifu error info: 0x301d9cb9cee80, ccu error info: 0x286d209822450474, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30468f0, fixp_error1 info: 0xa7, fsmId:0, tslot:0, thread:0, ctxid:0, blk:14, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 49, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x6008929018, mte error info: 0xa7030488f0, ifu error info: 0x1022489822a00, ccu error info: 0xa329b0c5104a0b0, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30488f0, fixp_error1 info: 0xa7, fsmId:0, tslot:0, thread:0, ctxid:0, blk:16, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 0, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x6113b70094, mte error info: 0xa7030538f0, ifu error info: 0x7e30407598880, ccu error info: 0x20380bcf24130aa1, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30538f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:17, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 2, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x7006dd6fa4, mte error info: 0xa7030488f0, ifu error info: 0x45ffdb5ba09c0, ccu error info: 0xb09c2c003281f732, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30488f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:19, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 3, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x73067a9cc2, mte error info: 0xa70307b8f0, ifu error info: 0x420316ac78880, ccu error info: 0x122610c5874a243, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x307b8f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:20, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 5, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x7217d10e6b, mte error info: 0xa70304a8f0, ifu error info: 0x72fe73378080, ccu error info: 0x1e2dcd065560d417, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x304a8f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:22, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 6, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x6812d40693, mte error info: 0xa70305f8f0, ifu error info: 0x70b1f909af240, ccu error info: 0x87229e8d705eb3c3, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x305f8f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:23, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 8, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x7b11d90fee, mte error info: 0xa7030488f0, ifu error info: 0x20f9c1c3308c0, ccu error info: 0x15c4c83e1680df1b, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30488f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:25, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 11, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0xe115512a1d, mte error info: 0xa7030488f0, ifu error info: 0x7358505f2f800, ccu error info: 0x220001c38000083, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30488f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:28, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 12, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0xc8180c1778, mte error info: 0xa70307a8f0, ifu error info: 0x5ce1df848c180, ccu error info: 0x83ec0ab87c911c11, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x307a8f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:29, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 13, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0xb218388218, mte error info: 0xa7030488f0, ifu error info: 0x3551e00e0c080, ccu error info: 0x348124019802bf3, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30488f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:30, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 14, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0xc913e1d936, mte error info: 0xa70304b8f0, ifu error info: 0x4332a8ba08980, ccu error info: 0x946c204e0fe12425, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x304b8f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:31, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 15, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x7b1bd41e2d, mte error info: 0xa7030468f0, ifu error info: 0x6ef5e1f353e00, ccu error info: 0xb93e00e157385, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30468f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:32, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 16, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x5906d551be, mte error info: 0xa7030578f0, ifu error info: 0x95ecf2d80480, ccu error info: 0x437e1306002000f0, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30578f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:33, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 17, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x5f01940a08, mte error info: 0xa7030488f0, ifu error info: 0x59ce83502d6c0, ccu error info: 0x2b73690f520005fa, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30488f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:34, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 18, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x6e02f4630e, mte error info: 0xa70304d8f0, ifu error info: 0x71351db007cc0, ccu error info: 0x4029330d09ffe0fc, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x304d8f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:35, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 22, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x610970f429, mte error info: 0xa7030598f0, ifu error info: 0x48625b143c80, ccu error info: 0xc49220c05b21f21, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30598f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:39, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 24, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x3019ef2614, mte error info: 0xa7030498f0, ifu error info: 0x2aadfc087c400, ccu error info: 0x8cd0811002010050, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30498f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:41, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 25, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x600d17a2ac, mte error info: 0xa7030468f0, ifu error info: 0x7f033b9e7fc00, ccu error info: 0x3ee2cfe45107438c, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30468f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:42, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 26, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x601112bb24, mte error info: 0xa7030498f0, ifu error info: 0x64d88008c03c0, ccu error info: 0xe4542213608f4000, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30498f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:43, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 30, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0xf104b7de09, mte error info: 0xa7030468f0, ifu error info: 0x27ee07f2019c0, ccu error info: 0x163ee9f37b430043, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30468f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:47, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 35, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0xd10616a270, mte error info: 0xa70304c8f0, ifu error info: 0x750027343f700, ccu error info: 0xc2583a0b78e5881f, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x304c8f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:2, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 36, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x71039c3806, mte error info: 0xa7030468f0, ifu error info: 0xfaa6d3242240, ccu error info: 0x2f52332f628d81dc, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30468f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:3, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 37, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x171147020a, mte error info: 0xa70304a8f0, ifu error info: 0x457b987cc0080, ccu error info: 0x3629c0b87e341bc9, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x304a8f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:4, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 38, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x590eb456b6, mte error info: 0xa70305d8f0, ifu error info: 0x30ee90f052580, ccu error info: 0x81e0a64511458109, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x305d8f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:5, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 40, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x63059b2104, mte error info: 0xa70304e8f0, ifu error info: 0x63bcd0bb8dc80, ccu error info: 0x2f3678243fe61019, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x304e8f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:7, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 41, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x781af5bb2c, mte error info: 0xa70304a8f0, ifu error info: 0x99141fa30280, ccu error info: 0x705ca50e407719b4, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x304a8f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:8, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 46, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x2117d773f4, mte error info: 0xa7030498f0, ifu error info: 0x6f14103fc7dc0, ccu error info: 0x63ed4adc48000266, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30498f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:13, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The error from device(chipId:3, dieId:0), serial number is 22, there is an exception of aivec error, core id is 48, error code = 0x800000, dump info: pc start: 0x124600c69b84, current: 0x124600c6a0f8, vec error info: 0x5103432438, mte error info: 0xa7030468f0, ifu error info: 0x1207fe07c0880, ccu error info: 0x60e3f83e2e1fa0d7, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c18003e400.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:333]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30468f0, fixp_error1 info: 0xa7, fsmId:1, tslot:0, thread:0, ctxid:0, blk:15, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_core_proc.cc][LINE:353]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] Kernel task happen error, retCode=0x31, [vector core exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1555]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] AIV Kernel happen error, retCode=0x31.[FUNC:GetError][FILE:stream.cc][LINE:1191]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1191]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] Aicore kernel execute failed, device_id=3, stream_id=2, report_stream_id=2, task_id=6393, flip_num=110, fault kernel_name=ScatterNdUpdate_e1b0109dfcfe48c3210f1a593265d1dc_high_precision_880015000, fault kernel info ext=none, program id=310, hash=14158041811606132296.[FUNC:GetError][FILE:stream.cc][LINE:1191]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] rtStreamSynchronizeWithTimeout execute failed, reason=[vector core exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
(Worker_TP3 pid=221636) ERROR 11-20 02:28:06 [multiproc_executor.py:671] synchronize stream with timeout failed, runtime result = 507035[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162]