A tensor with all NaNs was produced in VAE.

[bjornlarssen]

Why does this sometimes happen? I've been playing with the same prompt all afternoon, then went away for a while, came back, restarted the notebook and…

Traceback (most recent call last):
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/call_queue.py", line 56, in f
    res = list(func(*args, **kwargs))
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/call_queue.py", line 37, in f
    res = func(*args, **kwargs)
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/txt2img.py", line 52, in txt2img
    processed = process_images(p)
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/processing.py", line 480, in process_images
    res = process_images_inner(p)
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/processing.py", line 613, in process_images_inner
    devices.test_for_nans(x, "vae")
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/devices.py", line 136, in test_for_nans
    raise NansException(message)
modules.devices.NansException: A tensor with all NaNs was produced in VAE.

This happened before and I resolved it by deleting everything and reinstalling it. However, I don't want to update AUTOMATIC1111 again now that it finally started working :P I commented !git pull and I'm using the same notebook, model, prompts, settings as two hours ago. Can this be cache of some sort? I gave up on custom VAEs, it's set to "Automatic." A Colab problem of some sort? I don't even know where to begin to look. I googled the error and got results unrelated to Stable Diffusion, talking generally about neural networks.

[TheLastBen]

nothing changed in the last two hours

[bjornlarssen]

Any idea what could be causing this error? It's intermittent, random, and I can't reproduce it. Well, sometimes I can, as evidenced ;) Sometimes it happens five times in a row, then it doesn't for a while. I thought getting rid of AUTOMATIC updates would fix that. (Maybe there is another update request I'm missing…)

[bjornlarssen]

Just happened again. I restarted, because the image loading kept glitching, reused the prompt/model/etc., and got lots of NaNs in my VAE.

[bjornlarssen]

I keep seeing "runtime disconnected" as the notebook is executing. I think Colab is having problems. Will report back tomorrow ;)

[bjornlarssen]

It worked for a while last night. I'm trying inpainting now and keep getting the same error every time.

[TheLastBen]

do you have a custom vae somewhere ?
also for testing purposes you can run the colab without the gdrive cell to test the notebook