Add experimental Intel XPU backend support

[a86582751]

Summary

Adds experimental Intel XPU backend discovery and memory-management support so SeedVR2 nodes can load on PyTorch XPU systems.

Changes:

• Detect torch.xpu devices and expose them as xpu:0, xpu:1, etc. in SeedVR2 device dropdowns.
• Report XPU as the active backend from get_gpu_backend().
• Use system memory as a conservative shared-memory proxy for XPU/iGPU memory checks.
• Add XPU memory usage, cache clearing, and peak reset hooks when available.
• Add a small Transformers 5.x flash-attn mapping guard so optional flash-attn checks fail closed instead of aborting import when flash-attn is not installed.

Local validation

Tested on Windows portable ComfyUI with PyTorch 2.12.0+xpu on Intel Core Ultra x7 358H / Intel XPU.

Validated:

• ComfyUI extension entrypoint loads successfully.
• Node schemas build for:
  • SeedVR2VideoUpscaler
  • SeedVR2LoadDiTModel
  • SeedVR2LoadVAEModel
  • SeedVR2TorchCompileSettings
• Device list returns ['xpu:0'] and offload list returns ['none', 'cpu', 'xpu:0'].
• Inference smoke tests completed with:
  • DiT: seedvr2_ema_7b-Q8_0.gguf
  • VAE: ema_vae_fp16.safetensors
  • device: xpu:0
  • attention: sdpa
  • single image 768x768 -> 1080x1080
  • single image 1280x720 -> 3640x2048 with tiled VAE encode/decode

This is intended as experimental support. It does not add CUDA-only attention backends for XPU.

[a86582751]

Additional local validation logs from Intel XPU testing:

Smoke test: 768x768 -> 1080x1080

Environment:

• Windows portable ComfyUI
• PyTorch 2.12.0+xpu
• Intel Core Ultra x7 358H / Intel XPU
• DiT: seedvr2_ema_7b-Q8_0.gguf
• VAE: ema_vae_fp16.safetensors
• Device: xpu:0
• Attention: sdpa

Key log excerpt:

[INFO] got prompt


   ███████╗███████╗███████╗██████╗ ██╗   ██╗██████╗     ██████╗       ███████╗
   ██╔════╝██╔════╝██╔════╝██╔══██╗██║   ██║██╔══██╗    ╚════██╗      ██╔════╝
   ███████╗█████╗  █████╗  ██║  ██║██║   ██║██████╔╝     █████╔╝      ███████╗
   ╚════██║██╔══╝  ██╔══╝  ██║  ██║╚██╗ ██╔╝██╔══██╗    ██╔═══╝       ╚════██║
   ███████║███████╗███████╗██████╔╝ ╚████╔╝ ██║  ██║    ███████╗  ██╗ ███████║
   ╚══════╝╚══════╝╚══════╝╚═════╝   ╚═══╝  ╚═╝  ╚═╝    ╚══════╝  ╚═╝ ╚══════╝
   v2.5.24                                    © ByteDance Seed · NumZ · AInVFX
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[00:45:50.342] 🔧 Validating ema_vae_fp16.safetensors...
[00:45:51.007] 🏃 Creating new runner: DiT=seedvr2_ema_7b-Q8_0.gguf, VAE=ema_vae_fp16.safetensors
[00:45:51.083] 🚀 Creating DiT model structure on meta device
[00:45:51.452] 🎨 Creating VAE model structure on meta device
[00:45:52.626]
[00:45:52.626] 🎬 Starting upscaling generation...
[00:45:52.626] 🎬   Input: 1 frame, 768x768px → Padded: 1088x1088px → Output: 1080x1080px (shortest edge: 1080px)
[00:45:52.626] 🎬   Batch size: 1, Seed: 42, Channels: RGB
[00:45:52.626]
[00:45:52.626]  ━━━━━━━━ Phase 1: VAE encoding ━━━━━━━━
[00:45:52.627] 🎨 Materializing VAE weights to XPU:0: D:\ComfyUI_windows_portable\ComfyUI\models\SEEDVR2\ema_vae_fp16.safetensors
[00:45:53.297] 🎨 Encoding batch 1/1
[00:45:53.298] 📹   Sequence of 1 frames
[00:45:57.445]
[00:45:57.445]  ━━━━━━━━ Phase 2: DiT upscaling ━━━━━━━━
[00:45:57.449] 🚀 Materializing DiT weights to XPU:0: D:\ComfyUI_windows_portable\ComfyUI\models\SEEDVR2\seedvr2_ema_7b-Q8_0.gguf
[00:46:14.271] 🎬 Upscaling batch 1/1
EulerSampler: 100%|██████████████████████████████████████████████████████████████████████| 1/1 [00:10<00:00, 10.96s/it]
[00:46:32.616]
[00:46:32.617]  ━━━━━━━━ Phase 3: VAE decoding ━━━━━━━━
[00:46:32.618] 🔧 Pre-allocating output tensor: 1 frames, 1080x1080px, RGB (0.01GB)
[00:46:32.621] 🎨 Decoding batch 1/1
[00:46:36.864]
[00:46:36.865]  ━━━━━━━━ Phase 4: Post-processing ━━━━━━━━
[00:46:36.865] 📹 Post-processing batch 1/1
[00:46:36.919] 📹   Applying LAB perceptual color transfer
[00:46:37.964] 🎬 Output assembled: 1 frames, Resolution: 1080x1080px, Channels: RGB
[00:46:37.967]
[00:46:37.971] ✅ Upscaling completed successfully!
[00:46:42.433] ⚡ Average FPS: 0.02 frames/sec
[00:46:42.433]
[00:46:42.433]  ────────────────────────
[00:46:42.434] 💬 Questions? Updates? Watch, star & sponsor if you can!
[00:46:42.434] 🎬 https://www.youtube.com/@AInVFX
[00:46:42.434] ⭐💝 https://github.com/numz/ComfyUI-SeedVR2_VideoUpscaler
[INFO] Prompt executed in 52.41 seconds
[INFO] got prompt

Larger image test: 1280x720 -> 3640x2048

This run also exercised RGBA input and VAE tiled encode/decode.

Key log excerpt:

[INFO] got prompt


   ███████╗███████╗███████╗██████╗ ██╗   ██╗██████╗     ██████╗       ███████╗
   ██╔════╝██╔════╝██╔════╝██╔══██╗██║   ██║██╔══██╗    ╚════██╗      ██╔════╝
   ███████╗█████╗  █████╗  ██║  ██║██║   ██║██████╔╝     █████╔╝      ███████╗
   ╚════██║██╔══╝  ██╔══╝  ██║  ██║╚██╗ ██╔╝██╔══██╗    ██╔═══╝       ╚════██║
   ███████║███████╗███████╗██████╔╝ ╚████╔╝ ██║  ██║    ███████╗  ██╗ ███████║
   ╚══════╝╚══════╝╚══════╝╚═════╝   ╚═══╝  ╚═╝  ╚═╝    ╚══════╝  ╚═╝ ╚══════╝
   v2.5.24                                    © ByteDance Seed · NumZ · AInVFX
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[01:02:29.338] 🏃 Creating new runner: DiT=seedvr2_ema_7b-Q8_0.gguf, VAE=ema_vae_fp16.safetensors
[01:02:29.372] 🚀 Creating DiT model structure on meta device
[01:02:29.746] 🎨 Creating VAE model structure on meta device
[01:02:29.931]
[01:02:29.932] 🎬 Starting upscaling generation...
[01:02:29.932] 🎬   Input: 1 frame, 1280x720px → Padded: 3648x2048px → Output: 3640x2048px (shortest edge: 2048px, max edge: 4096px)
[01:02:29.933] 🎬   Batch size: 5, Seed: 2908158546, Channels: RGBA
[01:02:29.933]
[01:02:29.933]  ━━━━━━━━ Phase 1: VAE encoding ━━━━━━━━
[01:02:29.934]
[01:02:29.934] 💡 Tip: For 1 frames, batch_size=1 matches video length optimally
[01:02:29.934] 💡   Matching batch_size to shot length improves temporal coherence
[01:02:29.935]
[01:02:29.935] 🎨 Materializing VAE weights to XPU:0: D:\ComfyUI_windows_portable\ComfyUI\models\SEEDVR2\ema_vae_fp16.safetensors
[01:02:31.632] 🎨 Encoding batch 1/1
[01:02:31.632] 📹   Sequence of 1 frames
[01:02:31.671] 🎨   Using VAE tiled encoding (Tile: (1024, 1024), Overlap: (128, 128))
[01:02:44.591]
[01:02:44.592]  ━━━━━━━━ Phase 2: DiT upscaling ━━━━━━━━
[01:02:44.595] 🚀 Materializing DiT weights to XPU:0: D:\ComfyUI_windows_portable\ComfyUI\models\SEEDVR2\seedvr2_ema_7b-Q8_0.gguf
[01:03:19.661] 🎬 Upscaling batch 1/1
EulerSampler: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:35<00:00, 35.03s/it]
[01:04:03.065]
[01:04:03.079]  ━━━━━━━━ Phase 3: VAE decoding ━━━━━━━━
[01:04:03.082] 🔧 Pre-allocating output tensor: 1 frames, 3640x2048px, RGBA (0.06GB)
[01:04:03.093] 🎨 Decoding batch 1/1
[01:04:03.143] 🎨   Using VAE tiled decoding (Tile: (1024, 1024), Overlap: (128, 128))
[01:04:24.787]
[01:04:24.788]  ━━━━━━━━ Phase 4: Post-processing ━━━━━━━━
[01:04:24.788] 👻 Processing Alpha batch 1/1
[01:04:25.511] 📹 Post-processing batch 1/1
[01:04:25.553] 📹   Applying LAB perceptual color transfer
[01:04:26.566] 🎬 Output assembled: 1 frames, Resolution: 3640x2048px, Channels: RGBA
[01:04:26.567]
[01:04:26.575] ✅ Upscaling completed successfully!
[01:04:32.018] ⚡ Average FPS: 0.01 frames/sec
[01:04:32.019]
[01:04:32.020]  ────────────────────────
[01:04:32.020] 💬 Questions? Updates? Watch, star & sponsor if you can!
[01:04:32.021] 🎬 https://www.youtube.com/@AInVFX
[01:04:32.021] ⭐💝 https://github.com/numz/ComfyUI-SeedVR2_VideoUpscaler
[INFO] Prompt executed in 124.65 seconds

These are not benchmark claims, just reproducibility evidence that the XPU device path can load the nodes, materialize both VAE and DiT on XPU:0, and complete real SeedVR2 inference without CUDA.

[a86582751]

Update pushed in c22ce16.

Notes for review:

• The Intel XPU compatibility refinements in this PR were generated with Codex.
• The implementation was informed by local Windows portable ComfyUI testing and by Intel's llm-scaler Omni patch set: https://github.com/intel/llm-scaler/tree/main/omni/patches
• I adapted only the XPU-safe compatibility pieces from the Omni approach, including XPU-aware tensor offload checks, torchvision resize fallback settings, XPU output tensor handling, safetensors CPU-intermediate fallback for XPU allocation errors, and more complete XPU memory cleanup paths.
• I intentionally did not copy the broad XPU force-conversion of NaDiT parameters/buffers to compute dtype, because this fork is also being tested with GGUF Q8 weights and that conversion could inflate quantized weights in memory.

Latest local validation:

• Windows portable ComfyUI
• PyTorch 2.12.0+xpu
• Intel XPU device: xpu:0 / Intel Arc B390 GPU on Core Ultra x7 358H setup
• DiT: seedvr2_ema_7b-Q8_0.gguf
• VAE: ema_vae_fp16.safetensors
• Attention: sdpa
• Input: 1 RGBA frame, 1280x720
• Output: 3640x2048, tiled VAE encode/decode
• Result: completed successfully in 110.58s

[a86582751]

Updated this PR with a new experimental Intel Omni XPU attention backend for SeedVR2.

What changed in the PR branch:

• Added attention_mode=omni_xpu for SeedVR2 DiT attention.
• Probes omni_xpu_kernel.sdp and falls back to PyTorch SDPA when unavailable or when the shape is unsupported.
• Logs which attention backend is actually used and why a fallback happens.
• Keeps sdpa as the safe default.
• Updates README / CLI docs to mention omni_xpu.

Dependency note:

This mode depends on Intel's Omni work from llm-scaler, especially:

• https://github.com/intel/llm-scaler/tree/main/omni/ComfyUI-OmniXPU
• omni_xpu_kernel.sdp

The SeedVR2-side change does not vendor Intel's ComfyUI monkey patches into this PR. Users need to install ComfyUI-OmniXPU and a matching omni_xpu_kernel wheel separately, or just use the default sdpa path.

Local validation:

• Windows portable ComfyUI
• PyTorch 2.12.0+xpu
• Intel Core Ultra X7 358H / Arc B390 style XPU
• SeedVR2 3B Q8 GGUF + ema_vae_fp16.safetensors
• Z-Image-Turbo Q8 -> SeedVR2 upscale workflow
• omni_xpu produced visually comparable output to SDPA in local testing, with first-run timing improving from about 166s to about 150s in that workflow.

Known notes from testing:

• omni_xpu fast path is intentionally conservative: XPU tensor, fp16/bf16, head_dim 64/128, no dropout, no causal mask, no custom softmax scale, and matching q/k/v packed sequence lengths.
• Some 7B internal window/masked attention paths still use PyTorch SDPA; that is expected.
• For Intel's global ComfyUI-OmniXPU monkey patches, OMNIXPU_NORM=0 is recommended for now. In local Z-Image testing, the native Omni RMSNorm path could crash in SYCL/C++; SeedVR2 omni_xpu attention does not require the norm patch.
• When monkey-patching already-imported modules, avoid getattr(mod, name, None) on transformers lazy modules; checking mod.__dict__ avoids noisy lazy imports.
• Be careful with globally patching comfy.rmsnorm.rms_norm, because it affects functional RMSNorm call sites outside SeedVR2.
• CacheDiT is not recommended for low-step Z-Image-Turbo workflows in this setup: aggressive settings degraded quality, conservative defaults produced 0 cache hits, and torch.compile could recompile around CacheDiT's Python state counters.

Prebuilt wheels:

I intentionally did not add binary wheels to this upstream PR. For users of my Intel XPU fork, I put community-built Windows CPython 3.13 omni_xpu_kernel wheels on the fork main branch under wheels/:

https://github.com/a86582751/ComfyUI-SeedVR2_VideoUpscaler-intel-xpu/tree/main/wheels

Those include ptl_h, xe3_lpg, xe2_lpg, lnl_m, and bmg targets with notes about device matching.

[tinafengfun]

can you help to merge this pr as we need this for end customer evaluation? @numz