Accelerate Inference Section of Doc Broken

[AbhinavGopal]

Describe the bug

https://huggingface.co/docs/diffusers/en/tutorials/fast_diffusion

` from diffusers import StableDiffusionXLPipeline
import torch

pipe = StableDiffusionXLPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.bfloat16
).to("cuda")

prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
image = pipe(prompt, num_inference_steps=30).images[0]`

Results in:
Loading pipeline components...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:09<00:00, 1.35s/it] Traceback (most recent call last): File "/app/o.py", line 9, in <module> image = pipe(prompt, num_inference_steps=30).images[0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/kong/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/kong/lib/python3.11/site-packages/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py", line 1094, in __call__ ) = self.encode_prompt( ^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/kong/lib/python3.11/site-packages/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py", line 410, in encode_prompt prompt_embeds = text_encoder(text_input_ids.to(device), output_hidden_states=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/kong/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/kong/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/kong/lib/python3.11/site-packages/transformers/models/clip/modeling_clip.py", line 1490, in forward text_embeds = self.text_projection(pooled_output) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/kong/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/kong/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/kong/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 125, in forward return F.linear(input, self.weight, self.bias) RuntimeError: expected mat1 and mat2 to have the same dtype, but got: c10::Half != c10::BFloat16

Reproduction

`diffusers import StableDiffusionXLPipeline
import torch

pipe = StableDiffusionXLPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.bfloat16
).to("cuda")

prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
image = pipe(prompt, num_inference_steps=30).images[0]`

Logs

System Info

• 🤗 Diffusers version: 0.33.0.dev0
• Platform: Linux-5.10.234-225.895.amzn2.x86_64-x86_64-with-glibc2.31
• Running on Google Colab?: No
• Python version: 3.11.9
• PyTorch version (GPU?): 2.6.0+cu124 (True)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Huggingface_hub version: 0.29.1
• Transformers version: 4.49.0
• Accelerate version: 1.4.0
• PEFT version: 0.14.0
• Bitsandbytes version: 0.45.3
• Safetensors version: 0.5.3
• xFormers version: 0.0.29.post3
• Accelerator: NVIDIA L40S, 46068 MiB
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

Who can help?

@yiyixuxu @asomoza @sayakpaul

[hlky]

Hi @AbhinavGopal

Looks like this bug was introduced in transformers==4.49.0 and specifically when passed torch_dtype is not the original, here text_encoder_2 is in torch.float16 while other models (including text_encoder) are torch.bfloat16 which causes the error.

The issue appears to have been fixed in transformers main, and I've confirmed 4.48.0 is working. I'd recommend either temporarily downgrading to 4.48.0 or installing transformers from main (until 4.50.0 release).

[AbhinavGopal]

Thanks @hlky! Appreciate the response