Forge Train --> RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

[Steed9746]

Hi,

i´m using A1111 Forge and it works great so far.
Tried to test the Train function to create an embedding from pictures, but get this error message.
Quiet sure the "bug" is me, course i don´t know how to handle this well enough 😄
Does anyone how to solve this and maybe recommend a good tutorial on creating embedding?

And yes, i did a google search. Find a lot about that problem correlated with python. But not with SD.

Thank a lot in advance!

Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: f0.0.17v1.8.0rc-latest-276-g29be1da7
Commit hash: 29be1da
Launching Web UI with arguments:
Total VRAM 8192 MB, total RAM 32690 MB
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 3060 Ti : native
Hint: your device supports --pin-shared-memory for potential speed improvements.
Hint: your device supports --cuda-malloc for potential speed improvements.
Hint: your device supports --cuda-stream for potential speed improvements.
VAE dtype: torch.bfloat16
CUDA Stream Activated: False
Using pytorch cross attention
ControlNet preprocessor location: C:\ai\forge\webui\models\ControlNetPreprocessor
Loading weights [15012c538f] from C:\ai\forge\webui\models\Stable-diffusion\realisticVisionV51_v51VAE.safetensors
2024-03-17 10:06:41,981 - ControlNet - INFO - ControlNet UI callback registered.
model_type EPS
UNet ADM Dimension 0
Running on local URL: http://127.0.0.1:7861

To create a public link, set share=True in launch().
Startup time: 44.8s (prepare environment: 15.5s, import torch: 8.0s, import gradio: 2.9s, setup paths: 7.6s, initialize shared: 0.4s, other imports: 5.9s, load scripts: 3.1s, create ui: 0.8s, gradio launch: 0.4s).
Using pytorch attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using pytorch attention in VAE
extra {'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_l.logit_scale'}
To load target model SD1ClipModel
Begin to load 1 model
[Memory Management] Current Free GPU Memory (MB) = 7141.9814453125
[Memory Management] Model Memory (MB) = 454.2076225280762
[Memory Management] Minimal Inference Memory (MB) = 1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) = 5663.773822784424
Moving model(s) has taken 0.18 seconds
Model loaded in 6.1s (load weights from disk: 0.6s, forge instantiate config: 0.3s, forge load real models: 4.4s, calculate empty prompt: 0.8s).
Calculating sha256 for C:\ai\forge\webui\embeddings\sa1.pt: ab55eec1962e2846ac9886786d76875273e61dde7f83463d49cd8c1dbd88e267
Training at rate of 0.005 until step 100000
Preparing dataset...
0%| | 0/15 [00:00<?, ?it/s]To load target model AutoencoderKL
Begin to load 1 model
[Memory Management] Current Free GPU Memory (MB) = 6763.03662109375
[Memory Management] Model Memory (MB) = 159.55708122253418
[Memory Management] Minimal Inference Memory (MB) = 1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) = 5579.479539871216
Moving model(s) has taken 0.06 seconds
100%|██████████████████████████████████████████████████████████████████████████████████| 15/15 [00:01<00:00, 8.71it/s]
0%| | 0/100000 [00:00<?, ?it/s]*** Error training embedding
Traceback (most recent call last):
File "C:\ai\forge\webui\modules\textual_inversion\textual_inversion.py", line 547, in train_embedding
loss = shared.sd_model.forward(x, cond)[0] / gradient_step
File "C:\ai\forge\webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 846, in forward
return self.p_losses(x, c, t, *args, **kwargs)
File "C:\ai\forge\webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 885, in p_losses
x_noisy = self.q_sample(x_start=x_start, t=t, noise=noise)
File "C:\ai\forge\webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 356, in q_sample
return (extract_into_tensor(self.sqrt_alphas_cumprod, t, x_start.shape) * x_start +
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

---

[freecoderwaifu]

You should post Forge related issues on the Forge repo.

Use Kohya's GUI for all your training needs, plenty of Youtube tutorials out there.
https://github.com/bmaltais/kohya_ss

[Steed9746]

thanks a lot, will give it a try!

[dilectiogames]

use kohya_ss (is a different installation similar on how you installed automatic) that one is for training

[Steed9746]

thanks a lot, will give it a try!