FluxFillPipeline With Model Quantized By Bitsandbytes Has A Matrix Shapes Cannot Be Multiplied Bug #10833
Description
Describe the bug
Inferring with FluxFillPipeline quantized by bitsandbytes will cause a RuntimeError: mat1 and mat2 shapes cannot be multiplied .
Everything works fine when using FluxPipeline, FluxImg2ImgPipeline and FluxInpaintPipeline.
Reproduction
import torch
from diffusers import FluxFillPipeline
from diffusers.utils import load_image
image = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/cup.png")
mask = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/cup_mask.png")
pipe = FluxFillPipeline.from_pretrained("eramth/flux-4bit",torch_dtype=torch.float16).to("cuda")
pipe.enable_vae_tiling()
image = pipe(
prompt="a white paper cup",
image=image,
mask_image=mask,
height=1632,
width=1232,
max_sequence_length=512,
generator=torch.Generator("cpu").manual_seed(0)
).images[0]Logs
...
/usr/local/lib/python3.11/dist-packages/bitsandbytes/autograd/_functions.py in forward(ctx, A, B, out, bias, quant_state)
460 # 1. Dequantize
461 # 2. MatmulnN
--> 462 output = torch.nn.functional.linear(A, F.dequantize_4bit(B, quant_state).to(A.dtype).t(), bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (7854x384 and 64x3072)System Info
- 🤗 Diffusers version: 0.32.2
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Running on Google Colab?: Yes
- Python version: 3.11.11
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Flax version (CPU?/GPU?/TPU?): 0.10.3 (gpu)
- Jax version: 0.4.33
- JaxLib version: 0.4.33
- Huggingface_hub version: 0.28.1
- Transformers version: 4.48.3
- Accelerate version: 1.3.0
- PEFT version: 0.14.0
- Bitsandbytes version: 0.45.2
- Safetensors version: 0.5.2
- xFormers version: not installed
- Accelerator: Tesla T4, 15360 MiB
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No