RuntimeError: CUDA error: out of memory

[mdicillo]

Hello, I've tried many different values for the training flags and have yet to complete a training before I get the error:

RuntimeError: CUDA error: out of memory

I have halved the values of "r" and "lora_alpha" in the file "zh_rap_lora_config" until I got to "1," I have changed the --precision flag to 16, I have changed the --num_workers to "4" and below, I have changed the --learning_rate to "0.00005," but I still keep getting the "out of memory" error.

I am running Windows 11 with an RTX 3060 which has 12GB of VRAM...

Are there any other tips and/or tricks for optimizing the memory for the training?

The dataset I am working with is 6.6MB and I used 13 audio files to create the dataset which amounts to 14.2MB...I know that's not a direct correlation to the training memory used, but this isn't a gigantic dataset, so I'm not sure what exactly is going on...

Any help would be appreciated!

[woct0rdho]

You can try my fork for training with low VRAM: https://github.com/woct0rdho/ACE-Step

It should work with 12 GB VRAM

[mdicillo]

Hey!!! Thank you so much for the reply!!! I literally just downloaded your fork last night after reading the Subreddits and other discussion threads where this is mentioned. Thank you for your hard work!!! It's actually running and working as we speak!

It looks like all of the flags available in the original repo are set by default in your version (--num_workers, --epochs, etc...)...can these be changed, or are these values integral to your version working?

Thank you again for your awesome work!!!

[woct0rdho]

It's ok to change the flags, and you may try a few times tuning them to get a good quality lora.

[mdicillo]

Thank you!!!