RuntimeWarning: invalid value encountered in cast images = (images * 255).round().astype("uint8") output black image

[pavankay]

Describe the bug

When running the stable-diffusion-2-1 I get a runtime warning "RuntimeWarning: invalid value encountered in cast
images = (images * 255).round().astype("uint8")" and the image output is black.

Reproduction

My code:
import torch
from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler

model_id = "stabilityai/stable-diffusion-2-1"

comment#Use the DPMSolverMultistepScheduler (DPM-Solver++) scheduler here instead
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe = pipe.to("cuda")

prompt = "a photo of an astronaut riding a horse on mars"
image = pipe(prompt).images[0]

image.save("astronaut_rides_horse.png")

I got it from here: https://huggingface.co/stabilityai/stable-diffusion-2-1

Logs

Output:
PS C:\AI\diffusion> & c:/AI/diffusion/.conda/python.exe c:/AI/diffusion/main.py
model_index.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 537/537 [00:00<00:00, 556kB/s]
C:\AI\diffusion\.conda\lib\site-packages\huggingface_hub\file_download.py:149: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\Users\user\.cache\huggingface\hub\models--stabilityai--stable-diffusion-2-1. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.
To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development
  warnings.warn(message)
(…)ature_extractor/preprocessor_config.json: 100%|████████████████████████████████████████████████████████████████████████| 342/342 [00:00<?, ?B/s]
tokenizer/tokenizer_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████| 824/824 [00:00<?, ?B/s]
tokenizer/special_tokens_map.json: 100%|██████████████████████████████████████████████████████████████████████████████████| 460/460 [00:00<?, ?B/s] 
tokenizer/merges.txt: 100%|█████████████████████████████████████████████████████████████████████████████████████| 525k/525k [00:00<00:00, 29.9MB/s] 
scheduler/scheduler_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████| 345/345 [00:00<?, ?B/s] 
tokenizer/vocab.json: 100%|███████████████████████████████████████████████████████████████████████████████████| 1.06M/1.06M [00:00<00:00, 23.0MB/s] 
text_encoder/config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 633/633 [00:00<?, ?B/s] 
vae/config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 611/611 [00:00<?, ?B/s] 
unet/config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 939/939 [00:00<?, ?B/s] 
diffusion_pytorch_model.safetensors: 100%|██████████████████████████████████████████████████████████████████████| 335M/335M [00:11<00:00, 29.8MB/s] 
model.safetensors: 100%|██████████████████████████████████████████████████████████████████████████████████████| 1.36G/1.36G [00:31<00:00, 42.7MB/s] 
diffusion_pytorch_model.safetensors: 100%|████████████████████████████████████████████████████████████████████| 3.46G/3.46G [01:17<00:00, 44.7MB/s] 
Fetching 13 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 13/13 [01:18<00:00,  6.07s/it] 
Loading pipeline components...: 100%|████████████████████████████████████████████████████████████████████████████████| 6/6 [00:02<00:00,  2.71it/s] 
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [04:57<00:00,  5.95s/it]
C:\AI\diffusion\.conda\lib\site-packages\diffusers\image_processor.py:90: RuntimeWarning: invalid value encountered in cast
  images = (images * 255).round().astype("uint8")
PS C:\AI\diffusion> ^C
PS C:\AI\diffusion>

System Info

• diffusers version: 0.26.0
• Platform: Windows-10-10.0.22621-SP0
• Python version: 3.10.13
• PyTorch version (GPU?): 2.1.0+cu121 (True)
• Huggingface_hub version: 0.20.3
• Transformers version: 4.37.2
• Accelerate version: 0.26.1
• xFormers version: not installed
• Using GPU in script?: true NVIDIA GeForce GTX 1660 TI 6gb of vram
• Using distributed or parallel set-up in script?: I dont understand this one

Who can help?

No response

[Bhavay-2001]

Hi @sayakpaul @yiyixuxu, I would like to work on this issue. Please let me know how can I start. Thanks

[CTimmerman]

Same on my GTX 1660 Ti with CUDA 12.3.

Replace torch.float16 with torch.float32 to fix.

Same for webui using: webui.bat --no-half

Before:

D:\code\AI\Python>nvidia-smi
Sun Feb  4 16:18:35 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 546.12                 Driver Version: 546.12       CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                     TCC/WDDM  | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1660 Ti   WDDM  | 00000000:01:00.0  On |                  N/A |
| N/A   50C    P8               7W /  80W |   1371MiB /  6144MiB |      7%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      9940    C+G   ...r and Support Assistant\DSATray.exe    N/A      |
|    0   N/A  N/A     33880    C+G   ...werToys\PowerToys.PowerLauncher.exe    N/A      |
|    0   N/A  N/A     90960      C   ...rograms\Python\Python310\python.exe    N/A      |
|    0   N/A  N/A    109548    C+G   ...werToys\PowerToys.ColorPickerUI.exe    N/A      |
|    0   N/A  N/A    128552    C+G   ...PowerToys\PowerToys.PowerAccent.exe    N/A      |
+---------------------------------------------------------------------------------------+

d:\code\AI\Python>python du_lora.py
C:\Users\C\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\utils\outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
Loading pipeline components...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:11<00:00,  1.68s/it]
The config attributes {'skip_prk_steps': True} were passed to LCMScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
  0%|                                                                                                                                                                              | 0/8 [00:00<?, ?it/s]C:\Users\C\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\models\attention_processor.py:1244: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:263.)
  hidden_states = F.scaled_dot_product_attention(
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [08:23<00:00, 62.96s/it]
C:\Users\C\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\image_processor.py:90: RuntimeWarning: invalid value encountered in cast
  images = (images * 255).round().astype("uint8")

After:

d:\code\AI\Python>python du_lora.py
C:\Users\C\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\utils\outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
Loading pipeline components...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:06<00:00,  1.09it/s]
The config attributes {'skip_prk_steps': True} were passed to LCMScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
  0%|                                                                                                                                                                              | 0/8 [00:00<?, ?it/s]C:\Users\C\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\models\attention_processor.py:1244: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:263.)
  hidden_states = F.scaled_dot_product_attention(
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [48:16<00:00, 362.12s/it]
  0%|                                                                                                                                                                              | 0/8 [00:00<?, ?it/s]

Source:

# https://huggingface.co/nerijs/pixel-art-xl
import time
from diffusers import DiffusionPipeline, LCMScheduler
import torch

model_id = "stabilityai/stable-diffusion-xl-base-1.0"
lcm_lora_id = "latent-consistency/lcm-lora-sdxl"
pipe = DiffusionPipeline.from_pretrained(model_id, variant="fp16")
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)

pipe.load_lora_weights(lcm_lora_id, adapter_name="lora")
# Path not found? huggingface_hub.utils._errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-65bf9aae-1bf2b36602e3f1251ea11e7d;4af0f325-1b83-40e4-9cc1-262f5eb8f401)
pipe.load_lora_weights("pixel-art-xl.safetensors", adapter_name="pixel")

pipe.set_adapters(["lora", "pixel"], adapter_weights=[1.0, 1.2])
pipe.to(device="cuda", dtype=torch.float32)  # float16 results in black images after
# d:\code\AI\Python>python du_lora.py
# C:\Users\C\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\utils\outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
#   torch.utils._pytree._register_pytree_node(
# Loading pipeline components...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:11<00:00,  1.68s/it]
# The config attributes {'skip_prk_steps': True} were passed to LCMScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
#   0%|                                                                                                                                                                              | 0/8 [00:00<?, ?it/s]C:\Users\C\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\models\attention_processor.py:1244: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:263.)
#   hidden_states = F.scaled_dot_product_attention(
# 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [08:23<00:00, 62.96s/it]
# C:\Users\C\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\image_processor.py:90: RuntimeWarning: invalid value encountered in cast
#   images = (images * 255).round().astype("uint8")

prompt = "pixel, a cute corgi"
negative_prompt = "3d render, realistic"

num_images = 9

for i in range(num_images):
    img = pipe(
        prompt=prompt,
        negative_prompt=negative_prompt,
        num_inference_steps=8,
        guidance_scale=1.5,
    ).images[0]
    
    img.save(f"{time.strftime('%Y-%m-%d %H%M%S')}_lcm_lora_{i}.png")

[Bhavay-2001]

Hi @CTimmerman, does this problem exists for torch.float16 on a windows system? Should the solution be crafted? Thanks

[CTimmerman]

@Bhavay-2001 Yes:
Edition Windows 11 Home
Version 23H2
Installed on ‎28-‎11-‎2022
OS build 22631.3007
Experience Windows Feature Experience Pack 1000.22681.1000.0

[Bhavay-2001]

I think we need to discuss this with @sayakpaul @yiyixuxu and then I can contribute in this.

[CTimmerman]

Enabling autocast is better than raising the error to the user and outputting black images, but i think that suddenly stopped working in my webgui until i started using --no-half there.

[Bhavay-2001]

Soo how can we make this change? Any helps will be appreciated.

[sayakpaul]

We have made a number of fixes to the DPM schedulers. Could you try installing the latest version of diffusers and check again?

Cc: @yiyixuxu

[CTimmerman]

We have made a number of fixes to the DPM schedulers. Could you try installing the latest version of diffusers and check again?

Main doesn't look any different, and there are too many other branches to read. If this is not a problem limited to 16xx cards, then you can test it using my code by changing float32 back to float16.

I'm already using the main branch:

D:\code\AI\Python>python -m pip freeze
absl-py==2.1.0
accelerate==0.26.1
aiohttp==3.9.3
aiosignal==1.3.1
async-timeout==4.0.3
attrs==23.2.0
cachetools==5.3.2
certifi==2023.11.17
charset-normalizer==3.3.2
colorama==0.4.6
datasets==2.16.1
defusedxml==0.7.1
diffusers @ file:///D:/code/AI/Python/diffusers
dill==0.3.7
filelock==3.13.1
frozenlist==1.4.1
fsspec==2023.10.0
ftfy==6.1.3
google-auth==2.27.0
google-auth-oauthlib==1.2.0
grpcio==1.60.1
huggingface-hub==0.20.3
idna==3.6
importlib-metadata==7.0.1
Jinja2==3.1.3
keyboard==0.13.5
Markdown==3.5.2
MarkupSafe==2.1.4
mouse==0.7.1
mpmath==1.3.0
multidict==6.0.5
multiprocess==0.70.15
networkx==3.2.1
numpy==1.26.3
oauthlib==3.2.2
packaging==23.2
pandas==2.2.0
peft==0.7.0
pillow==10.2.0
protobuf==4.23.4
psutil==5.9.8
pyarrow==15.0.0
pyarrow-hotfix==0.6
pyasn1==0.5.1
pyasn1-modules==0.3.0
python-dateutil==2.8.2
pytz==2024.1
PyYAML==6.0.1
regex==2023.12.25
requests==2.31.0
requests-oauthlib==1.3.1
rsa==4.9
safetensors==0.4.2
six==1.16.0
sympy==1.12
tensorboard==2.15.1
tensorboard-data-server==0.7.2
tokenizers==0.15.1
torch==2.2.0+cu121
torchaudio==2.2.0+cu121
torchvision==0.17.0
tqdm==4.66.1
transformers==4.37.2
typing_extensions==4.9.0
tzdata==2023.4
urllib3==2.1.0
wcwidth==0.2.13
Werkzeug==3.0.1
xxhash==3.4.1
yarl==1.9.4
zipp==3.17.0

D:\code\AI\Python>python -m pip show diffusers
Name: diffusers
Version: 0.26.0.dev0
Summary: State-of-the-art diffusion in PyTorch and JAX.
Home-page: https://github.com/huggingface/diffusers
Author: The Hugging Face team (past and future) with the help of all our contributors (https://github.com/huggingface/diffusers/graphs/contributors)
Author-email: patrick@huggingface.co
License: Apache 2.0 License
Location: c:\users\c\appdata\local\programs\python\python310\lib\site-packages
Requires: filelock, huggingface-hub, importlib-metadata, numpy, Pillow, regex, requests, safetensors
Required-by:

D:\code\AI\Python>python -m pip install diffusers --upgrade
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: diffusers in c:\users\c\appdata\local\programs\python\python310\lib\site-packages (0.26.0.dev0)
Collecting diffusers
  Downloading diffusers-0.26.1-py3-none-any.whl.metadata (18 kB)
Requirement already satisfied: importlib-metadata in c:\users\c\appdata\local\programs\python\python310\lib\site-packages (from diffusers) (7.0.1)
Requirement already satisfied: filelock in c:\users\c\appdata\local\programs\python\python310\lib\site-packages (from diffusers) (3.13.1)
Requirement already satisfied: huggingface-hub>=0.20.2 in c:\users\c\appdata\local\programs\python\python310\lib\site-packages (from diffusers) (0.20.3)
Requirement already satisfied: numpy in c:\users\c\appdata\local\programs\python\python310\lib\site-packages (from diffusers) (1.26.3)
Requirement already satisfied: regex!=2019.12.17 in c:\users\c\appdata\local\programs\python\python310\lib\site-packages (from diffusers) (2023.12.25)
Requirement already satisfied: requests in c:\users\c\appdata\local\programs\python\python310\lib\site-packages (from diffusers) (2.31.0)
Requirement already satisfied: safetensors>=0.3.1 in c:\users\c\appdata\local\programs\python\python310\lib\site-packages (from diffusers) (0.4.2)
Requirement already satisfied: Pillow in c:\users\c\appdata\local\programs\python\python310\lib\site-packages (from diffusers) (10.2.0)
Requirement already satisfied: fsspec>=2023.5.0 in c:\users\c\appdata\local\programs\python\python310\lib\site-packages (from huggingface-hub>=0.20.2->diffusers) (2023.10.0)
Requirement already satisfied: tqdm>=4.42.1 in c:\users\c\appdata\local\programs\python\python310\lib\site-packages (from huggingface-hub>=0.20.2->diffusers) (4.66.1)
Requirement already satisfied: pyyaml>=5.1 in c:\users\c\appdata\local\programs\python\python310\lib\site-packages (from huggingface-hub>=0.20.2->diffusers) (6.0.1)
Requirement already satisfied: typing-extensions>=3.7.4.3 in c:\users\c\appdata\local\programs\python\python310\lib\site-packages (from huggingface-hub>=0.20.2->diffusers) (4.9.0)
Requirement already satisfied: packaging>=20.9 in c:\users\c\appdata\local\programs\python\python310\lib\site-packages (from huggingface-hub>=0.20.2->diffusers) (23.2)
Requirement already satisfied: zipp>=0.5 in c:\users\c\appdata\local\programs\python\python310\lib\site-packages (from importlib-metadata->diffusers) (3.17.0)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\c\appdata\local\programs\python\python310\lib\site-packages (from requests->diffusers) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in c:\users\c\appdata\local\programs\python\python310\lib\site-packages (from requests->diffusers) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\c\appdata\local\programs\python\python310\lib\site-packages (from requests->diffusers) (2.1.0)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\c\appdata\local\programs\python\python310\lib\site-packages (from requests->diffusers) (2023.11.17)
Requirement already satisfied: colorama in c:\users\c\appdata\local\programs\python\python310\lib\site-packages (from tqdm>=4.42.1->huggingface-hub>=0.20.2->diffusers) (0.4.6)
Downloading diffusers-0.26.1-py3-none-any.whl (1.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.9/1.9 MB 20.5 MB/s eta 0:00:00
Installing collected packages: diffusers
  Attempting uninstall: diffusers
    Found existing installation: diffusers 0.26.0.dev0
    Uninstalling diffusers-0.26.0.dev0:
      Successfully uninstalled diffusers-0.26.0.dev0
Successfully installed diffusers-0.26.1

D:\code\AI\Python>cd diffusers && git status
On branch main
Your branch is behind 'origin/main' by 1 commit, and can be fast-forwarded.
  (use "git pull" to update your local branch)

nothing to commit, working tree clean

D:\code\AI\Python\diffusers>git pull
Updating 13001ee3..fbdf26ba
Fast-forward
 examples/dreambooth/train_dreambooth_lora_sdxl.py | 100 +++++++++++++++-------
 1 file changed, 67 insertions(+), 33 deletions(-)

That code is not used in this issue.

[pavankay]

how can I fix the problem though this has also been happening with the upscaler models too. When I run it in google colab it works.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sayakpaul]

Gently pinging @yiyixuxu.