[Guide] Training SDXL LoRAs with ROCm (AMD) in 2025

[TotallyTroll]

Been struggling for some time to train LoRAs on my AMD GPU (7900XTX).
Most of the time I spent trying to make it work with GUI version. No luck.
Also all the guides over the internet are for quite old version of everything (rocm, pytorch, etc.) that doesn't work anymore.

So here are quick commands and instructions that made it possible to train SDXL loras with sd-scripts in 2025.

0. At this point you should have AMD / HIP drivers installed.
1. Installation commands (Ubuntu 24.04)

git clone https://github.com/kohya-ss/sd-scripts.git
cd sd-scripts
python3.10 -m venv venv
source venv/bin/activate
pip install --upgrade pip wheel
pip install -r requirements.txt
pip uninstall torch torchvision
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.3
pip install tensorflow-rocm==2.17.0 -f https://repo.radeon.com/rocm/manylinux/rocm-rel-6.3/ --upgrade
# This is for my 7900XTX. Use 10.3.0 for previous GPUs.
export HSA_OVERRIDE_GFX_VERSION=11.0.0 
accelerate config
# This machine No distributed training, no, no, no, all, fp16
mkdir log reg train output models #to create folders
pip3 install onnxruntime-rocm -f https://repo.radeon.com/rocm/manylinux/rocm-rel-6.3.2

2. Tokenizer fix. Navigate to ./sd-scripts/library -> open sdxl_train_util.py
   Change TOKENIZER2_PATH to be the same as TOKENIZER1_PATH:

TOKENIZER1_PATH = "openai/clip-vit-large-patch14"
TOKENIZER2_PATH = "openai/clip-vit-large-patch14"

For some reason it can't find or download "laion/CLIP-ViT-bigG-14-laion2B-39B-b160k" tokenizer. So script fails all the time with Tokenizer error.

3. Prepare your training sdxl_lora_train.sh file (or whatever name you want).
   Here are params that I used and they worked.

export HSA_OVERRIDE_GFX_VERSION=11.0.0 #use different version for your AMD GPU

time accelerate launch sdxl_train_network.py \
--pretrained_model_name_or_path=/home/user/path_to_SDXL_checkpoint/sd_xl_base_1.0.safetensors \
--train_data_dir=/home/user/dataset_folder/ \ #at this location there should be another folder with name like 10_somedatasetname where 10 - stands for repeats per image. and 'somedatasetname' will be your trigger word.
--caption_extension=txt \
--output_dir=output \
--logging_dir=log \
--network_module=networks.lora \
--output_name=trained_lora_name \
--save_model_as=safetensors \
--resolution=1024 \
--train_batch_size=1 \
--max_train_epochs=30 \
--save_every_n_epochs=2 \
--seed=1111 \
--mixed_precision=fp16 \
--save_precision=fp16 \
--optimizer_type=AdamW \ # Adafactor also works  fine. I still didn't decide which one is better. AdamW8bit - won't work.
--learning_rate=1e-4 \
--lr_scheduler=constant \
--network_dim=64 \
--network_alpha=32 \
--cache_latents \
--cache_latents_to_disk \
--cache_text_encoder_outputs \
--cache_text_encoder_outputs_to_disk \
--no_half_vae \
--max_token_length=225 \
--min_bucket_reso=256 \
--max_bucket_reso=2048 \
--bucket_no_upscale \
--min_snr_gamma=5 \
--multires_noise_iterations=10 \
--multires_noise_discount=0.1 \
--noise_offset=0.03 \
--network_train_unet_only \
--persistent_data_loader_workers \
--max_data_loader_n_workers=1 \
--sdpa

4. chmod +x sdxl_lora_train.sh
   And run it: ./sdxl_lora_train.sh

I still don't like end results much. Since same dataset and parameters trained at tensor.art - provided much better LoRA.
At least now I have an opportunity to test different parameters on my local machine and maybe some day achieve proper results with AMD GPU.

[ddpasa]

Thanks for sharing your config! I started a thread here to share our configs and get feedback!

One thing caught my eye. You are using Adafactor with the default parameters. This means that --learning_rate=1e-4 will likely be ignored, as the default behaviour in Adafactor is to use an adaptive LR.

[TotallyTroll]

One thing caught my eye. You are using Adafactor with the default parameters. This means that --learning_rate=1e-4 will likely be ignored, as the default behaviour in Adafactor is to use an adaptive LR.

I'm quite bad at understanding those parameters. It was just a big deal to make it work with AMD GPU finally.

During last couple of days, since original post - I've been playing with this parameters a bit and seems AdamW is better for my case at least (I don't create characters or all that kind of scenarios people typically train loras for).
So, --learning_rate=1e-4 param at least worked for AdamW.

P.S. updated my original post with less confusing config, considering your advice!