Support for INT8, MXFP8 and NVFP4 models - #613
Conversation
Add the official Comfy-Org/SeedVR2 fp16 and fp8_e4m3fn checkpoints (3B, 7B, 7B-sharp) to the model registry so they appear in the DiT loader dropdown and auto-download on first use. These carry the same 635 NaDiT tensors as the existing numz weights (plus baked-in conditioning tensors, ignored by the non-strict load_state_dict), so they are drop-in compatible. Adds a ModelInfo.subfolder field and threads it into the HuggingFace download URL, since Comfy-Org stores weights under diffusion_models/. The on-disk filename stays flat; existing numz/AInVFX entries are unaffected. Quantized formats (int8_convrot, mxfp8, nvfp4) are intentionally deferred to a follow-up that adds comfy_kitchen dequantization support. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…on-load
The official Comfy-Org *_int8_convrot.safetensors models store each quantized
Linear as int8 weight + fp32 per-channel weight_scale + a comfy_quant JSON blob
({"format":"int8_tensorwise","convrot":true,"convrot_groupsize":256}).
Plain nn.Linear cannot consume these.
Add _dequantize_comfy_quant_state() in model_loader, invoked at the end of
load_quantized_state_dict. It reconstructs the ComfyUI QuantizedTensor exactly
as comfy.ops._load_quantized_module does (TensorWiseINT8Layout with convrot) and
dequantizes back to fp16, so the existing NaDiT model loads the weights
unchanged. This is universal — it works on the eager path and does not require
the cu130 INT8 tensor-core kernels — at the cost of true quantized-inference
VRAM/speed savings.
Registers the 3B/7B/7B-sharp int8_convrot models. Non-quantized fp16/fp8/GGUF
checkpoints pass through as a no-op. Packed nvfp4 and block-scaled mxfp8 raise a
clear NotImplementedError for now.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend _dequantize_comfy_quant_state to the remaining comfy_kitchen formats:
- mxfp8: fp8_e4m3 weight + uint8 block scales (viewed as e8m0fnu), full shape.
- nvfp4: uint8 weight packed 2x fp4/byte + fp8_e4m3 block scale + fp32
tensor-wise scale; logical in-dim is weight.shape[1]*2.
Both are reconstructed as comfy QuantizedTensors (TensorCoreMXFP8Layout /
TensorCoreNVFP4Layout) exactly as comfy.ops._load_quantized_module and
dequantized to fp16 on load, verified against real layers. Registers the
3B/7B/7B-sharp mxfp8 and nvfp4 models, completing coverage of the Comfy-Org
diffusion_models repo.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Hi @Joly0 Thank you for your work. So, what was the speed difference in the videos? Did you test the difference between int8 and NVFP4, GGUF, and FP16? Was there a difference in output quality? The images show different results; the actual tests should be done on the videos to reveal the true results. |
|
Hey, so I did not test with Videos, just static images, but performance was pretty equal, I would say int8 was slightly faster than fp8. Regarding quality I couldn't notice any drop in quality for int8 compared to fp8 or gguf, there may be a very very small drop compared to fp16, but that's to be expected. From my testing I would say that nvfp4 and mxfp8 are slightly worse then int8, maybe slightly below fp8, but no huge difference. But as I tested this with static images and not with Videos and with my limited hardware in mind, I would advice someone else should maybe test and confirm everything is fine regarding quality and performance. |
2 participants
Also added the files from the official ComfyUI Huggingface page for the downloads https://huggingface.co/Comfy-Org/SeedVR2/tree/main/diffusion_models
Runs the int8_convrot version no problem:
Fixes #607