Possible to finetune model (like Waifu Diffusion, not dreambooth) with 12G VRAM?

[sgsdxzy]

It is possible to dreambooth with around 8G VRAM #3995 , so it's definitely doable on 12G using the Adam8bit implementation.
So my problem is, if I want to finetune the entire model, not just dreambooth with a specific subject, it is possible with 12G VRAM? Can we port/use the same optimization to model training?

[ClashSAN]

https://colab.research.google.com/drive/1vrh_MUSaAMaC5tsLWDxkFILKJ790Z4Bl?usp=sharing
maybe work on optimizing this one?

[sgsdxzy]

I seem to find it, I need to change every occurrence of torch.optim.AdamW in stable diffusion repo to bitsandbytes.optim.AdamW8bit. Don't know if it's necessary to change every occurrence of torch.optim.Adam.
I am going to test after I plug my 3080Ti back.