Good starting config for a PixArt Sigma fine tune?

[AndreRatzenberger]

Hey hey!

Before I waste precious compute figuring out a decent config for my first PixArt Sigma finetune, I thought perhaps you guys already have some experience you'd like to share.

The PixArt Alpha config seems viable... so I'll probably start with that. Alternatively, I might try CAME since the training scripts in PixArt's repos use that optimizer?!

How does PixArt react to learning rate, scheduling, and other important parameters compared to SDXL?

My dataset consists of 40k sexy images and is the base of one of the more popular NSFW SDXL models, so I hope I can recreate that success with PixArt as well, because that model needs more hype and love after all the SD3 drama.

Thanks!

[huagya]

I am training with CAME, using a learning rate of 1e-5, a batch size of 12, and a constant schedule, and the results are good.

However, since it may vary from person to person, I recommend searching for training results from predecessors on the onetrainer or Pixart Discord server and trying various things for yourself.
Some people on Civitai also share their training values.

You might need to change your approach compared to SDXL settings, but since the training speed is fast, once you find the right settings, the training itself should be easier.

[AndreRatzenberger]

This was my first try yesterday evening:

config_came_funny.json

But unfortunately, I woke up to all images being just plain bright green. It overburnt the overburn into a solid color, lol.

Did I do anything horrendous wrong? I used 2e-5; is it so fickle that being off by a factor of 2 means RIP run?

[huagya]

I checked the settings, but I don't think there's anything unusual.
I am also training with 2e-5 for 30,000 steps, and I have never experienced a collapse...
I am training with bf16, so there might be a difference in learning rate when using float32.

I will share my settings. They might not be ideal, but it could be helpful to see if someone else's settings cause a collapse. Batch size can be adjusted to fit your VRAM, as it has no significant impact.

Please try clearing the cache and then testing.
It might also be good to start with a small dataset to explore the settings.

Although it may not be related to the collapse issue, I am training with a dataset of 70,000 images.

sigma_1024px.json

[AndreRatzenberger]

bf16 changed everything. Wow, I just started a new quick run with 2k images to get a feel for the whole model and process, and after two hours, it already looks better than training SDXL for a day, lol.

Thanks!

[RexGrey]

It seems that this model is slow to train, I'm using a 4090 with training set to batch 18 and it's about 8-9s/it, slower than even sdxl and it's also impossible to adjust the text encoder