Failed while running with GPU

[andreclaudino]

I ran using just the CPU, to improve performance, I wish to run using GPU, but received the following error:

Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run _EagerConst: Dst tensor is not initialized.

The complete message is:

File "/home/claudino/Projetos/OpenSource/stable-diffusion-tensorflow/stable_diffusion_tf/stable_diffusion.py", line 270, in get_models
    diffusion_model.load_weights(diffusion_model_weights_fpath)
  File "/home/claudino/miniconda3/envs/stable-diffusion/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/claudino/miniconda3/envs/stable-diffusion/lib/python3.10/site-packages/keras/backend.py", line 4302, in batch_set_value
    x.assign(np.asarray(value, dtype=dtype_numpy(x)))
tensorflow.python.framework.errors_impl.InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run _EagerConst: Dst tensor is not initialized.

Cant determine the cause.

My environment:

(stable-diffusion) $> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:        22.04
Codename:       jamm

(stable-diffusion) $> nvidia-smi 
Tue Jan 17 17:31:00 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.60.13    Driver Version: 525.60.13    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| N/A   56C    P8     7W /  N/A |    208MiB /  6144MiB |     35%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1572      G   /usr/lib/xorg/Xorg                109MiB |
|    0   N/A  N/A      3293    C+G   ...014073573827879945,131072       96MiB |
+-----------------------------------------------------------------------------+

Cuda 11.2, tensorflow 2.10.0, cudnn 8.1.0

[alfiq]

Hello Andre,
what are your computer specs? (VRAM, RAM, CPU, GPU), Iv'e encountered this error when my VRAM is low.

[andreclaudino]

@alfiq, 15Gi Ram + 15 Gi Swap, processor is a core i7 9th gen, GPU is a GTC 1660 Ti with 6Gb