Masked training LoRA newbie questions

[SamTyurenkov]

Hi everyone,

here is my masked dataset:

I am trying to create a LoRA that will generate a well positioned side cap on the girls head.
LoRA is trained for the 3d render SDXL model, also im only training unet, cause im short on VRAM, can it provide good results?

I named concept "military scout hat"

So far Im getting an underfit samples - subject either not present at all, or its an unrecognizable mix type of deformed hat.

• How should I caption a masked image? only describe whats in the mask or not?
• How much should I mask? I took the subject and just a little bit of the surroundings, so it understands how I want it placed on the head, is it correct?
• Any other tips and advice please?

Thanks!

Update:
Here is the samples I get now, they either underfit, or overfit right away, also all the other parts of the image are affected too much:

[O-J1]

Please read MX's reply: #347 (comment)

[SamTyurenkov]

Hi, yeah I saw it, but it wasnt clear about captioning and how big to expand the mask outside of the trained subject. Also if feathering should be used.

[O-J1]

Hi, yeah I saw it, but it wasnt clear about captioning and how big to expand the mask outside of the trained subject. Also if feathering should be used.

I think you have failed to fully understand what he said then 😅 Masks are not a replacement for cropping when necessary.

Mask should have 64px gap from the intended object. You should caption only what is in the mask. Do not set the weight rediculously low or you will have whack outcomes as per MX's comment and crucially get more data, all of your pics are pretty much all the same.

If you need more help join the discord