[Bug]: [NPU]Qwen-Image-2512 use usp failed in 1328*1328

[zyqzhang1996]

Your current environment

The output of python collect_env.py

800T A2

Your code version

The commit id or version of vllm-omni

commit id: 3fc4f988eabf562572821896847dc96b69a4cf74

🐛 Describe the bug

also occurred on 15841056 and10561584

export ASCEND_RT_VISIBLE_DEVICES=4,5
export HCCL_OP_EXPANSION_MODE="AIV"

python /vllm-workspace/vllm-omni/examples/offline_inference/text_to_image/text_to_image.py \
  --model /home/weights/Qwen-Image-2512/ \
  --prompt "a cup of coffee on the table" \
  --seed 42 \
  --cfg_scale 4.0 \
  --num_images_per_prompt 1 \
  --num_inference_steps 50 \
  --height 1328 \
  --width 1328 \
  --output outputs/coffee.png \
  --cache_backend cache_dit \
  --ulysses_degree 2 \

log:

[rank0]:[E119 16:32:47.013946230 compiler_depend.ts:444] operator():build/CMakeFiles/torch_npu.dir/compiler_depend.ts:1042 NPU function error: call aclnnFlashAttentionScore failed, error code is 561103
[ERROR] 2026-01-19-16:32:47 (PID:10443, Device:0, RankID:0) ERR00100 PTA call acl api failed.
EZ9999: Inner Error!
EZ9999[PID: 10443] 2026-01-19-16:32:47.486.947 (EZ9999):  get unsupported atten_mask shape, the shape is [1, 6902]. B=[1], N=[12], Sq=[6902], Skv=[6902], supported atten_mask shape can be [B, N, Sq, Skv], [B, 1, Sq, Skv], [1, 1, Sq, Skv] and [Sq, Skv].[FUNC:AnalyzeOptionalInput][FILE:flash_attention_score_tiling_general.cpp][LINE:1621]
        TraceBack (most recent call last):
       fail to analyze context info.[FUNC:GetShapeAttrsInfo][FILE:flash_attention_score_tiling_general.cpp][LINE:866]
       Tiling failed
       Tiling Failed.
       Kernel Run failed. opType: 38, FlashAttentionScore
       launch failed for FlashAttentionScore, errno:561103.

Exception raised from operator() at build/CMakeFiles/torch_npu.dir/compiler_depend.ts:1042 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xb0 (0xffffa92848c0 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x68 (0xffffa922c140 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0x111d6b4 (0xfffdf972d6b4 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0x29f0894 (0xfffdfb000894 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x9cc700 (0xfffdf8fdc700 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x9cd2dc (0xfffdf8fdd2dc in /usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x9cb1f8 (0xfffdf8fdb1f8 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0xd29cc (0xffffb6e629cc in /lib/aarch64-linux-gnu/libstdc++.so.6)
frame #8: <unknown function> + 0x80398 (0xffffb7040398 in /lib/aarch64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0xe9e9c (0xffffb70a9e9c in /lib/aarch64-linux-gnu/libc.so.6)

[rank1]:[E119 16:32:47.014632430 compiler_depend.ts:444] operator():build/CMakeFiles/torch_npu.dir/compiler_depend.ts:1042 NPU function error: call aclnnFlashAttentionScore failed, error code is 561103
[ERROR] 2026-01-19-16:32:47 (PID:10444, Device:1, RankID:1) ERR00100 PTA call acl api failed.
EZ9999: Inner Error!
EZ9999[PID: 10444] 2026-01-19-16:32:47.487.817 (EZ9999):  get unsupported atten_mask shape, the shape is [1, 6902]. B=[1], N=[12], Sq=[6902], Skv=[6902], supported atten_mask shape can be [B, N, Sq, Skv], [B, 1, Sq, Skv], [1, 1, Sq, Skv] and [Sq, Skv].[FUNC:AnalyzeOptionalInput][FILE:flash_attention_score_tiling_general.cpp][LINE:1621]
        TraceBack (most recent call last):
       fail to analyze context info.[FUNC:GetShapeAttrsInfo][FILE:flash_attention_score_tiling_general.cpp][LINE:866]
       Tiling failed
       Tiling Failed.
       Kernel Run failed. opType: 38, FlashAttentionScore
       launch failed for FlashAttentionScore, errno:561103.

Exception raised from operator() at build/CMakeFiles/torch_npu.dir/compiler_depend.ts:1042 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xb0 (0xffffaf0b48c0 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x68 (0xffffaf05c140 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0x111d6b4 (0xfffdff98d6b4 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0x29f0894 (0xfffe01260894 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x9cc700 (0xfffdff23c700 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x9cd2dc (0xfffdff23d2dc in /usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x9cb1f8 (0xfffdff23b1f8 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0xd29cc (0xffffbcca29cc in /lib/aarch64-linux-gnu/libstdc++.so.6)
frame #8: <unknown function> + 0x80398 (0xffffbce80398 in /lib/aarch64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0xe9e9c (0xffffbcee9e9c in /lib/aarch64-linux-gnu/libc.so.6)

[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270] Error executing RPC: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is aclnnFlashAttentionScore.
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270] [ERROR] 2026-01-19-16:32:47 (PID:10443, Device:0, RankID:0) ERR00100 PTA call acl api failed.
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270] Traceback (most recent call last):
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/workspace/vllm-omni/vllm_omni/diffusion/worker/gpu_diffusion_worker.py", line 265, in execute_rpc
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     result = func(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]              ^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/workspace/vllm-omni/vllm_omni/diffusion/worker/npu/npu_worker.py", line 118, in generate
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     return self.execute_model(requests, self.od_config)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     return func(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]            ^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/workspace/vllm-omni/vllm_omni/diffusion/worker/npu/npu_worker.py", line 138, in execute_model
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     output = self.pipeline.forward(req)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]              ^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/workspace/vllm-omni/vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py", line 771, in forward
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     latents = self.diffuse(
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]               ^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/workspace/vllm-omni/vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py", line 610, in diffuse
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     noise_pred = self.transformer(
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]                  ^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     return self._call_impl(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     return forward_call(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/cache_dit/caching/cache_adapters/cache_adapter.py", line 439, in new_forward_with_hf_hook
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     outputs = new_forward(self, *args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/cache_dit/caching/cache_adapters/cache_adapter.py", line 427, in new_forward
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     outputs = original_forward(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/workspace/vllm-omni/vllm_omni/diffusion/models/qwen_image/qwen_image_transformer.py", line 906, in forward
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     encoder_hidden_states, hidden_states = block(
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]                                            ^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     return self._call_impl(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     return forward_call(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/cache_dit/caching/cache_blocks/pattern_base.py", line 321, in forward
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     ) = self.call_Mn_blocks(  # middle
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/cache_dit/caching/cache_blocks/pattern_base.py", line 448, in call_Mn_blocks
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     hidden_states = block(
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]                     ^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     return self._call_impl(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     return forward_call(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/workspace/vllm-omni/vllm_omni/diffusion/models/qwen_image/qwen_image_transformer.py", line 653, in forward
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     attn_output = self.attn(
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]                   ^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     return self._call_impl(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     return forward_call(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/workspace/vllm-omni/vllm_omni/diffusion/models/qwen_image/qwen_image_transformer.py", line 439, in forward
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     if hidden_states_mask is not None and hidden_states_mask.all():
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is aclnnFlashAttentionScore.
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270] [ERROR] 2026-01-19-16:32:47 (PID:10443, Device:0, RankID:0) ERR00100 PTA call acl api failed.
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270] Error executing RPC: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is aclnnFlashAttentionScore.
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270] [ERROR] 2026-01-19-16:32:47 (PID:10444, Device:1, RankID:1) ERR00100 PTA call acl api failed.
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270] Traceback (most recent call last):
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/workspace/vllm-omni/vllm_omni/diffusion/worker/gpu_diffusion_worker.py", line 265, in execute_rpc
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     result = func(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]              ^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/workspace/vllm-omni/vllm_omni/diffusion/worker/npu/npu_worker.py", line 118, in generate
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     return self.execute_model(requests, self.od_config)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     return func(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]            ^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/workspace/vllm-omni/vllm_omni/diffusion/worker/npu/npu_worker.py", line 138, in execute_model
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     output = self.pipeline.forward(req)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]              ^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/workspace/vllm-omni/vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py", line 771, in forward
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     latents = self.diffuse(
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]               ^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/workspace/vllm-omni/vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py", line 610, in diffuse
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     noise_pred = self.transformer(
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]                  ^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     return self._call_impl(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     return forward_call(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/cache_dit/caching/cache_adapters/cache_adapter.py", line 439, in new_forward_with_hf_hook
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     outputs = new_forward(self, *args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/cache_dit/caching/cache_adapters/cache_adapter.py", line 427, in new_forward
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     outputs = original_forward(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/workspace/vllm-omni/vllm_omni/diffusion/models/qwen_image/qwen_image_transformer.py", line 906, in forward
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     encoder_hidden_states, hidden_states = block(
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]                                            ^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     return self._call_impl(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     return forward_call(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/cache_dit/caching/cache_blocks/pattern_base.py", line 321, in forward
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     ) = self.call_Mn_blocks(  # middle
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/cache_dit/caching/cache_blocks/pattern_base.py", line 448, in call_Mn_blocks
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     hidden_states = block(
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]                     ^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     return self._call_impl(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     return forward_call(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/workspace/vllm-omni/vllm_omni/diffusion/models/qwen_image/qwen_image_transformer.py", line 653, in forward
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     attn_output = self.attn(
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]                   ^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     return self._call_impl(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     return forward_call(*args, **kwargs)
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]   File "/workspace/vllm-omni/vllm_omni/diffusion/models/qwen_image/qwen_image_transformer.py", line 439, in forward
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]     if hidden_states_mask is not None and hidden_states_mask.all():
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is aclnnFlashAttentionScore.
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270] [ERROR] 2026-01-19-16:32:47 (PID:10444, Device:1, RankID:1) ERR00100 PTA call acl api failed.
[Stage-0] ERROR 01-19 16:32:47 [gpu_diffusion_worker.py:270]
[Stage-0] ERROR 01-19 16:32:47 [diffusion_engine.py:187] Generation failed: 'dict' object has no attribute 'error'
INFO 01-19 16:32:47 [log_utils.py:550] {'type': 'request_level_metrics',
INFO 01-19 16:32:47 [log_utils.py:550]  'request_id': '0_4c569671-ae5b-4ee4-a2c3-6a661d0a4f58',
INFO 01-19 16:32:47 [log_utils.py:550]  'e2e_time_ms': 189.84675407409668,
INFO 01-19 16:32:47 [log_utils.py:550]  'e2e_tpt': 0.0,
INFO 01-19 16:32:47 [log_utils.py:550]  'e2e_total_tokens': 0,
INFO 01-19 16:32:47 [log_utils.py:550]  'transfers_total_time_ms': 0.0,
INFO 01-19 16:32:47 [log_utils.py:550]  'transfers_total_bytes': 0,
INFO 01-19 16:32:47 [log_utils.py:550]  'stages': {0: {'stage_gen_time_ms': 183.68911743164062,
INFO 01-19 16:32:47 [log_utils.py:550]                 'num_tokens_out': 0,
INFO 01-19 16:32:47 [log_utils.py:550]                 'num_tokens_in': 0}}}
Processed prompts: 100%|███████████████████████████████████████| 1/1 [00:00<00:00,  5.28img/s, est. speed stage-0 img/s: 0.00, avg e2e_lat: 0.0ms]
INFO 01-19 16:32:47 [omni.py:782] [Summary] {'e2e_requests': 1,| 1/1 [00:00<00:00,  5.29img/s, est. speed stage-0 img/s: 0.00, avg e2e_lat: 0.0ms]
INFO 01-19 16:32:47 [omni.py:782]  'e2e_total_time_ms': 191.65873527526855,
INFO 01-19 16:32:47 [omni.py:782]  'e2e_sum_time_ms': 189.84675407409668,
INFO 01-19 16:32:47 [omni.py:782]  'e2e_total_tokens': 0,
INFO 01-19 16:32:47 [omni.py:782]  'e2e_avg_time_per_request_ms': 189.84675407409668,
INFO 01-19 16:32:47 [omni.py:782]  'e2e_avg_tokens_per_s': 0.0,
INFO 01-19 16:32:47 [omni.py:782]  'wall_time_ms': 191.65873527526855,
INFO 01-19 16:32:47 [omni.py:782]  'final_stage_id': {'0_4c569671-ae5b-4ee4-a2c3-6a661d0a4f58': 0},
INFO 01-19 16:32:47 [omni.py:782]  'stages': [{'stage_id': 0,
INFO 01-19 16:32:47 [omni.py:782]              'requests': 1,
INFO 01-19 16:32:47 [omni.py:782]              'tokens': 0,
INFO 01-19 16:32:47 [omni.py:782]              'total_time_ms': 190.59062004089355,
INFO 01-19 16:32:47 [omni.py:782]              'avg_time_per_request_ms': 190.59062004089355,
INFO 01-19 16:32:47 [omni.py:782]              'avg_tokens_per_s': 0.0}],
INFO 01-19 16:32:47 [omni.py:782]  'transfers': []}
Adding requests:   0%|                                                                                                      | 0/1 [00:00<?, ?it/s]
[Stage-0] INFO 01-19 16:32:47 [omni_stage.py:677] Received shutdown signal
[Stage-0] INFO 01-19 16:32:47 [gpu_diffusion_worker.py:304] Worker 1: Received shutdown message
[Stage-0] INFO 01-19 16:32:47 [gpu_diffusion_worker.py:304] Worker 0: Received shutdown message
[Stage-0] INFO 01-19 16:32:47 [gpu_diffusion_worker.py:325] event loop terminated.
[Stage-0] INFO 01-19 16:32:47 [gpu_diffusion_worker.py:325] event loop terminated.
[Stage-0] INFO 01-19 16:32:47 [npu_worker.py:251] Worker 0: Shutdown complete.
[Stage-0] INFO 01-19 16:32:47 [npu_worker.py:251] Worker 1: Shutdown complete.
Total generation time: 5.1984 seconds (5198.37 ms)
INFO 01-19 16:32:52 [text_to_image.py:168] Outputs: [OmniRequestOutput(request_id='', finished=True, stage_id=0, final_output_type='image', request_output=[namespace(request_id='0_4c569671-ae5b-4ee4-a2c3-6a661d0a4f58', output=None)], images=[], prompt=None, latents=None, metrics={})]
Traceback (most recent call last):
  File "/vllm-workspace/vllm-omni/examples/offline_inference/text_to_image/text_to_image.py", line 198, in <module>
    main()
  File "/vllm-workspace/vllm-omni/examples/offline_inference/text_to_image/text_to_image.py", line 177, in main
    raise ValueError("Invalid request_output structure or missing 'images' key")
ValueError: Invalid request_output structure or missing 'images' key
[ERROR] 2026-01-19-16:32:52 (PID:10169, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ZJY0516]

@gcanlin It's npu specific bug

[gcanlin]

Got it. We haven't supported 2512 on NPU yet. Welcome to contribute. We will support it in a few days.

[muziyuhui666]

Qwen-Image-2512 does not feature any updates to the structural code compared with Qwen-Image. Since Qwen-Image has already been adapted for NPU, this model can also run on NPU as well.
This error is likely caused by not using the latest version of the MindIE-SD library. It is recommended to compile and install MindIE-SD from the source code: https://gitcode.com/Ascend/MindIE-SD.
If you encounter errors during the installation process, you may refer to: #871.

[zyqzhang1996]

I upgraded MindIE-SD, but it still reports an error. @muziyuhui666

......[Stage-0] INFO 01-21 09:47:31 [shm_broadcast.py:501] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
............[Stage-0] INFO 01-21 09:48:31 [shm_broadcast.py:501] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
............[Stage-0] INFO 01-21 09:49:31 [shm_broadcast.py:501] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
......WARNING 01-21 09:50:00 [omni.py:326] [Orchestrator] Initialization timeout: 0/1 stages ready. Missing stages: [0]
ERROR 01-21 09:50:00 [omni.py:340] [Orchestrator] Stage initialization failed. Troubleshooting Steps:
ERROR 01-21 09:50:00 [omni.py:340]   1) Verify GPU/device assignment in config (runtime.devices) is correct.
ERROR 01-21 09:50:00 [omni.py:340]   2) Check GPU/host memory availability; reduce model or batch size if needed.
ERROR 01-21 09:50:00 [omni.py:340]   3) Check model weights path and network reachability (if loading remotely).
ERROR 01-21 09:50:00 [omni.py:340]   4) Increase initialization wait time (stage_init_timeout or call-site timeout).

============================================================
Generation Configuration:
  Model: /home/weights/Qwen-Image-2512/
  Inference steps: 50
  Cache backend: cache_dit
  Parallel configuration: ulysses_degree=2, ring_degree=1
  Image size: 1328x1328
============================================================

Adding requests:   0%|                                                                                                      | 0/1 [00:00<?, ?it/s......[Stage-0] INFO 01-21 09:50:31 [shm_broadcast.py:501] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
('Warning: torch.save with "_use_new_zipfile_serialization = False" is not recommended for npu tensor, which may bring unexpected errors and hopefully set "_use_new_zipfile_serialization = True"', 'if it is necessary to use this, please convert the npu tensor to cpu tensor for saving')
Warning: The current version of the file storing weights is old, and it is relanded due to internal bug of torch and compatibility issue. We will deprecate the loading support for this type of file in the future, please use newer torch to re-store the weight file.
[Stage-0] INFO 01-21 09:50:33 [omni_stage.py:668] Max batch size: 1
[Stage-0] INFO 01-21 09:50:33 [omni_diffusion.py:115] Prepared 1 requests for generation.
('Warning: torch.save with "_use_new_zipfile_serialization = False" is not recommended for npu tensor, which may bring unexpected errors and hopefully set "_use_new_zipfile_serialization = True"', 'if it is necessary to use this, please convert the npu tensor to cpu tensor for saving')
[Stage-0] INFO 01-21 09:50:33 [cache_dit_backend.py:883] Refreshing cache context for transformer with num_inference_steps: 50
INFO 01-21 09:50:33 [cache_adapter.py:723] ✅ Refreshed cache context: transformer_blocks_281459272470864, Collected Context Config: DBCache_F1B0_W4I1M0MC3_R0.24_N50_CFG1, Calibrator Config: None
[Stage-0] INFO 01-21 09:50:33 [cache_dit_backend.py:883] Refreshing cache context for transformer with num_inference_steps: 50
INFO 01-21 09:50:33 [cache_adapter.py:723] ✅ Refreshed cache context: transformer_blocks_281459595375696, Collected Context Config: DBCache_F1B0_W4I1M0MC3_R0.24_N50_CFG1, Calibrator Config: None
[rank0]:[E121 09:50:33.053804220 compiler_depend.ts:444] operator():build/CMakeFiles/torch_npu.dir/compiler_depend.ts:1042 NPU function error: call aclnnFlashAttentionScore failed, error code is 561103
[ERROR] 2026-01-21-09:50:33 (PID:14109, Device:0, RankID:0) ERR00100 PTA call acl api failed.
EZ9999: Inner Error!
EZ9999[PID: 14109] 2026-01-21-09:50:33.527.083 (EZ9999):  get unsupported atten_mask shape, the shape is [1, 6902]. B=[1], N=[12], Sq=[6902], Skv=[6902], supported atten_mask shape can be [B, N, Sq, Skv], [B, 1, Sq, Skv], [1, 1, Sq, Skv] and [Sq, Skv].[FUNC:AnalyzeOptionalInput][FILE:flash_attention_score_tiling_general.cpp][LINE:1621]
        TraceBack (most recent call last):
       fail to analyze context info.[FUNC:GetShapeAttrsInfo][FILE:flash_attention_score_tiling_general.cpp][LINE:866]
       Tiling failed
       Tiling Failed.
       Kernel Run failed. opType: 38, FlashAttentionScore
       launch failed for FlashAttentionScore, errno:561103.

Exception raised from operator() at build/CMakeFiles/torch_npu.dir/compiler_depend.ts:1042 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xb0 (0xffff7cd748c0 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x68 (0xffff7cd1c140 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0x111d6b4 (0xfffdcd22d6b4 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0x29f0894 (0xfffdceb00894 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x9cc700 (0xfffdccadc700 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x9cd2dc (0xfffdccadd2dc in /usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x9cb1f8 (0xfffdccadb1f8 in /usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0xd29cc (0xffff8a9529cc in /lib/aarch64-linux-gnu/libstdc++.so.6)
frame #8: <unknown function> + 0x80398 (0xffff8ab30398 in /lib/aarch64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0xe9e9c (0xffff8ab99e9c in /lib/aarch64-linux-gnu/libc.so.6)

[gcanlin]

@zyqzhang1996 Hi, #1031 has fixed this bug. Could you merge it and try again?

[zyqzhang1996]

@gcanlin Hi, I cloned the main branch and update vllm_omni/diffusion/attention/backends/sdpa.py, commenting out fa3-fwd, then compiled vllm-omni and upgraded mindiesd.
However, when running inference, I encountered the following error:

[Stage-0] INFO 01-30 10:17:51 [platform.py:52] Defaulting to diffusion attention backend FLASH_ATTN
Traceback (most recent call last):
  File "/vllm-workspace/vllm-omni/vllm_omni/diffusion/attention/selector.py", line 45, in _load_backend_cls
    module = importlib.import_module(module_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.14/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/vllm-workspace/vllm-omni/vllm_omni/diffusion/attention/backends/flash_attn.py", line 27, in <module>
    raise ImportError(
ImportError: FlashAttentionBackend requires Flash Attention. Please install one of: fa3-fwd, flash-attention, or flash-attn. Otherwise, use SDPA backend by setting DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA

I tried to set export DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA, but encountered a different error. How should I resolve this?

[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315] Error executing RPC: SetPrecisionMode:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:175 NPU function error: at_npu::native::AclSetCompileopt(aclCompileOpt::ACL_PRECISION_MODE, precision_mode), error code is 500001
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315] [ERROR] 2026-01-30-10:24:03 (PID:1885, Device:1, RankID:1) ERR00100 PTA call acl api failed
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315] [Error]: The internal ACL of the system is incorrect.
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]         Rectify the fault based on the error information in the ascend log.
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315] [PID: 1885] 2026-01-30-10:24:02.994.354 Environment_Error_Import_Python_Module_Failed(EC0010): Failed to import Python module torch._inductor.exc.InductorError: KeyError: 'cpu'
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315] Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315] ..
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]         Solution: Check that all required components are properly installed and the specified Python path matches the Python installation directory. If the path does not match the directory, run set_env.sh in the installation package.
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]         TraceBack (most recent call last):
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]         get device uuid failed, deviceId = 1, runtime result = 207000[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]         Failed to import Python module torch._inductor.exc.InductorError: KeyError: 'cpu'
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315] Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315] ..
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]         AOE failed to call InitCannKB.[FUNC:Initialize][FILE:python_adapter_manager.cc][LINE:47]
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]         Failed to initialize TeConfigInfo.
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]         [GraphOpt][InitializeInner][InitTbeFunc] Failed to init tbe.[FUNC:InitializeTeFusion][FILE:tbe_op_store_adapter.cc][LINE:1966]
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]         [GraphOpt][InitializeInner][InitTeFusion]: Failed to initialize TeFusion.[FUNC:InitializeInner][FILE:tbe_op_store_adapter.cc][LINE:1933]
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]         [SubGraphOpt][PreCompileOp][InitAdapter] InitializeAdapter adapter [tbe_op_adapter] failed! Ret [4294967295][FUNC:InitializeAdapter][FILE:op_store_adapter_manager.cc][LINE:79]
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]         [SubGraphOpt][PreCompileOp][Init] Initialize op store adapter failed, OpsStoreName[tbe-custom].[FUNC:Initialize][FILE:op_store_adapter_manager.cc][LINE:120]
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]         [FusionMngr][Init] Op store adapter manager init failed.[FUNC:Initialize][FILE:fusion_manager.cc][LINE:117]
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]         PluginManager InvokeAll failed.[FUNC:Initialize][FILE:ops_kernel_manager.cc][LINE:83]
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]         OpsManager initialize failed.[FUNC:InnerInitialize][FILE:gelib.cc][LINE:222]
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]         GELib::InnerInitialize failed.[FUNC:Initialize][FILE:gelib.cc][LINE:159]
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]         GEInitialize failed.[FUNC:GEInitializeV2][FILE:ge_api_v2.cc][LINE:330]
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]         GEInitializeV2 initialize failed.[FUNC:GEInitialize][FILE:ge_api.cc][LINE:171]
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]         [Initialize][Ge]GEInitialize failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:147]
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]         [Init][Compiler]Init compiler failed[FUNC:ReportInnerError][FILE:acl_op_executor_log_inner.cpp][LINE:108]
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]         [Set][Options]OpCompileProcessor init failed![FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:131]
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315] Traceback (most recent call last):
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/vllm-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 310, in execute_rpc
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     result = func(*args, **kwargs)
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]              ^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/vllm-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 130, in generate
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     return self.execute_model(request, self.od_config)
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/vllm-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 152, in execute_model
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     return self.model_runner.execute_model(req)
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     return func(*args, **kwargs)
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/vllm-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_model_runner.py", line 174, in execute_model
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     output = self.pipeline.forward(req)
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]              ^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/vllm-workspace/vllm-omni/vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py", line 780, in forward
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     latents = self.diffuse(
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]               ^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/vllm-workspace/vllm-omni/vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py", line 610, in diffuse
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     noise_pred = self.transformer(
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]                  ^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     return self._call_impl(*args, **kwargs)
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     return forward_call(*args, **kwargs)
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/cache_dit/caching/cache_adapters/cache_adapter.py", line 448, in new_forward_with_hf_hook
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     outputs = new_forward(self, *args, **kwargs)
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/cache_dit/caching/cache_adapters/cache_adapter.py", line 436, in new_forward
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     outputs = original_forward(*args, **kwargs)
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/vllm-workspace/vllm-omni/vllm_omni/diffusion/models/qwen_image/qwen_image_transformer.py", line 955, in forward
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     encoder_hidden_states, hidden_states = block(
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]                                            ^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     return self._call_impl(*args, **kwargs)
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     return forward_call(*args, **kwargs)
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/cache_dit/caching/cache_blocks/pattern_base.py", line 250, in forward
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     hidden_states, encoder_hidden_states = self.call_Fn_blocks(
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]                                            ^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/cache_dit/caching/cache_blocks/pattern_base.py", line 426, in call_Fn_blocks
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     hidden_states = block(
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]                     ^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     return fn(*args, **kwargs)
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     return forward_call(*args, **kwargs)
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/vllm-workspace/vllm-omni/vllm_omni/diffusion/models/qwen_image/qwen_image_transformer.py", line 683, in forward
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     img_modulated, img_gate1 = self.img_norm1(hidden_states, img_mod1, modulate_index)
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     return self._call_impl(*args, **kwargs)
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     return forward_call(*args, **kwargs)
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/vllm-workspace/vllm-omni/vllm_omni/diffusion/layers/custom_op.py", line 32, in forward
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     return self._forward_method(*args, **kwargs)
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/vllm-workspace/vllm-omni/vllm_omni/diffusion/layers/adalayernorm.py", line 88, in forward_npu
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     output = torch_npu.npu_layer_norm_eval(
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]   File "/usr/local/python3.11.14/lib/python3.11/site-packages/torch/_ops.py", line 1255, in __call__
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]     return self._op(*args, **kwargs)
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 01-30 10:24:03 [diffusion_worker.py:315] RuntimeError: SetPrecisionMode:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:175 NPU function error: at_npu::native::AclSetCompileopt(aclCompileOpt::ACL_PRECISION_MODE, precision_mode), error code is 500001

[gcanlin]

Thanks for the feedback!

1. It looks like the selector has bug, which leads to route to fa3-fwd(CUDA) on NPU. Let me fix it.
2. Could you try adding the flag --enforce_eager? I will submit a PR to make it default.