Images are black when only using GPU (CUDA 11.7) along with NSFW warning

[1MochaChan1]

Describe the bug

I am new to AI in general, and I am trying to use the diffusers package by following the hugging face tutorial for running the model online and locally. But both of the methods (GPU) gives me a black image in return. However when I use the CPU mode pipe.to("cpu"), I get the results as expected.

In this case I am using the
CompVis/stable-diffusion-v1-4 model that I got from using the following snippet:

git lfs install
git clone https://huggingface.co/CompVis/stable-diffusion-v1-4

Reproduction

from diffusers import StableDiffusionPipeline
import torch


pipe = StableDiffusionPipeline.from_pretrained(
    "D:\\4_Projects\\9_AI\\stable-diffusion-v1-4",
    revision="fp16", 
    torch_dtype=torch.float16,
)

pipe = pipe.to("cuda")

prompt = "a photo of an astronaut riding a horse on mars"
pipe.enable_attention_slicing()
image = pipe(prompt).images[0]

image.save(f"{prompt}.png")

The fixes I tried:

1. Editing the safety_checker.py

Editing the line has_nsfw_concepts = [len(res["bad_concepts"]) > 0 for res in result] to has_nsfw_concepts = [False for res in result]

2. Using a different seed (1024, 3214)
3. Using a different scheduler (LMSDiscreteScheduler)

Logs

The following is the only things I got in the terminal using the local model.

100%|██████████████████████████████████████████████████████████████████████████| 51/51 [04:32<00:00,  5.33s/it] 
Potential NSFW content was detected in one or more images. A black image will be returned instead. Try again with a different prompt and/or seed.
PS D:\4_Projects\9_AI\text_to_image_python>

System Info

• diffusers version: 0.9.0
• Platform: Windows-10-10.0.19044-SP0
• Python version: 3.10.8
• PyTorch version (GPU?): 1.13.0 (True)
• Huggingface_hub version: 0.11.1
• Transformers version: 4.25.1
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: I do not understand what distributed or parallel setup could be in my script. However I've read an article where accelerate is used for distributed setup, and I do have accelerate package installed.

[averad]

@1MochaChan1 add safety_checker=None, to your pipe to disable the safety checker

from diffusers import StableDiffusionPipeline
import torch


pipe = StableDiffusionPipeline.from_pretrained(
    "D:\\4_Projects\\9_AI\\stable-diffusion-v1-4",
    revision="fp16", 
    torch_dtype=torch.float16,
    safety_checker=None,
)

pipe = pipe.to("cuda")

prompt = "a photo of an astronaut riding a horse on mars"
pipe.enable_attention_slicing()
image = pipe(prompt).images[0]

image.save(f"{prompt}.png")

[pcuenca]

I was going to suggest the same as @averad. @1MochaChan1, please let us know if that works :)

[1MochaChan1]

I was going to suggest the same as @averad. @1MochaChan1, please let us know if that works :)

Yes I tried this, and I also tried disabling the safety checker from safety_checker.py. It doesn't give me any warning, but again returns a black image.

This is how my code looks now

from diffusers import StableDiffusionPipeline, LMSDiscreteScheduler
import torch


pipe = StableDiffusionPipeline.from_pretrained(
    "D:\\4_Projects\\9_AI\\stable-diffusion-v1-4",
    revision="fp16", 
    torch_dtype=torch.float16,
    safety_checker = None
)

pipe = pipe.to("cuda")

prompt = "a photo of an astronaut riding a horse on mars"
pipe.enable_attention_slicing()
image = pipe(prompt).images[0]

image.save(f"{prompt}.png")

[pcuenca]

Hi @1MochaChan1,

Your code looks correct to me. Unfortunately, I don't have a Windows system to test right now, but the same code worked for me in Linux.

A couple things you could try:

• Verify that git lfs worked and your git clone has a reasonable size on disk. Mine takes about 12GB of space.
• Try without fp16 first.

In addition, inference seems slow in your system. What GPU are you using?

[1MochaChan1]

Hey @pcuenca thanks for responding,

• Verify that git lfs worked and your git clone has a reasonable size on disk. Mine takes about 12GB of space.
• Try without fp16 first.

1. Yes git lfs has worked correctly the cloned directory occupies 10.2 GB of disk space
2. I tried that, the code runs fine without any GPU memory allocation error. But the problem still persists (Black image returned on non NSFW prompts)

In addition, inference seems slow in your system. What GPU are you using?

I have a GTX 1650 4GB GDDR6 GPU (using a laptop Legion 5 15arh05)

[pcuenca]

Thanks for your answer. I have a feeling that something in that GPU architecture prevents it from running stable diffusion correctly. Someone recently posted something similar in a private (for now) fast.ai forum. They were using a GTX 1650 Ti and couldn't get it to work either.

I can't think of anything else to try here. Sorry about that. Please, do let us know if you are able to debug the problem further.

[1MochaChan1]

@pcuenca Yes, even I had this assumption, but since I didn't know a lot about AI and models I just thought of raising a bug.
Thanks a lot for your support though. I'll stick to CPU for now.

[palant]

Yes, I’m experiencing the same issue (on Linux) and it’s somehow related to the GPU. In my case it’s a GTX 1660 6GB, and printing out the “image” in numpy_to_pil shows that it’s all NaN values. Safety checker is not at fault here, disabling it doesn’t change anything. I might be able to investigate further and figure out where things go wrong, but running on CPU works.

[palant]

I managed to reduce the issue to the following code:

import torch
import numpy
from torch.nn import Conv2d

device = 'cuda'
data = []
for i in range(2):
    data.append([])
    for j in range(640):
        data[i].append([])
        for k in range(64):
            data[i][j].append([])
            for l in range(64):
                data[i][j][k].append(0.5)
tensor = torch.tensor(data, device=device, dtype=torch.float16)
conv = Conv2d(640, 640, 3, padding=1, device=device, dtype=torch.float16)

tensor = conv(tensor)
count = numpy.sum(numpy.isnan(tensor.detach().cpu().numpy()))
print(f'{count} NaN values')

The resulting tensor here contains more than a million NaN values. And for some reason it only happens for a particular tensor size range: 2x640x64x36 to 2x640x64x225 . Smaller or larger tensors have no NaN values in the result. And it only happens with exactly 640 channels, not 639 and not 641.

Calculating with float32 doesn’t appear to have this issue, at least I couldn’t find a tensor size that would similarly produce NaNs (I don’t have enough VRAM to run the entire calculation in float32).

So this is an upstream issue of course. But the question is still whether some parameters can be tweaked to work around it.

[1MochaChan1]

I managed to reduce the issue to the following code:

import torch
import numpy
from torch.nn import Conv2d

device = 'cuda'
data = []
for i in range(2):
    data.append([])
    for j in range(640):
        data[i].append([])
        for k in range(64):
            data[i][j].append([])
            for l in range(64):
                data[i][j][k].append(0.5)
tensor = torch.tensor(data, device=device, dtype=torch.float16)
conv = Conv2d(640, 640, 3, padding=1, device=device, dtype=torch.float16)

tensor = conv(tensor)
count = numpy.sum(numpy.isnan(tensor.detach().cpu().numpy()))
print(f'{count} NaN values')

The resulting tensor here contains more than a million NaN values. And for some reason it only happens for a particular tensor size range: 2x640x64x36 to 2x640x64x225 . Smaller or larger tensors have no NaN values in the result. And it only happens with exactly 640 channels, not 639 and not 641.

Calculating with float32 doesn’t appear to have this issue, at least I couldn’t find a tensor size that would similarly produce NaNs (I don’t have enough VRAM to run the entire calculation in float32).

So this is an upstream issue of course. But the question is still whether some parameters can be tweaked to work around it.

Thanks a lot for your contribution, I'll give this a try since I have a GTX 1650 with 4GB of RAM

[palant]

I looked through the memory optimizations on https://huggingface.co/docs/diffusers/optimization/fp16 and found that torch.backends.cudnn.benchmark = True fixed this, so now I can run on the GPU. The alternative is using a bunch more tweaks to bring memory usage further down and run with float32 which is considerably faster anyway for some reason.

[tongchangD]

@1MochaChan1 add safety_checker=None, to your pipe to disable the safety checker

from diffusers import StableDiffusionPipeline
import torch


pipe = StableDiffusionPipeline.from_pretrained(
    "D:\\4_Projects\\9_AI\\stable-diffusion-v1-4",
    revision="fp16", 
    torch_dtype=torch.float16,
    safety_checker=None,
)

pipe = pipe.to("cuda")

prompt = "a photo of an astronaut riding a horse on mars"
pipe.enable_attention_slicing()
image = pipe(prompt).images[0]

image.save(f"{prompt}.png")

good job