Question on "Contains_faces"

[giteeeeee]

Could you explain what does this new feature do?

Thanks

[TheLastBen]

Simply it applies prior preservation, but only on the text_encoder, still experimental, so feedback from users would help.

[DarkAlchy]

I use 30 images though.

[TheLastBen]

make it 20 for this time

[DarkAlchy]

One thing I still do not get about resuming. If I already trained with 10% then resume do I make that now 0% since it already trained 10%?

[TheLastBen]

just uncheck the box train_text_encoder, or set it to 0% it's the same thing

[DarkAlchy]

The result was absolutely horrid. I described it below to help. edit: I looked at the notebook but I couldn't figure out where can we set the text at manually instead of the percentage? We need a way to set that manually versus the percentage deal. Maybe I was 270 step for text for a 5k steps of training but with the percentage it becomes a hassle and I am not even sure it would allow 5.4% in the example.

[TheLastBen]

Getting great results with it though (1500 text_encoder, 3000 steps unet) :

[DarkAlchy]

You can do it as you want, I personally train without the text_enc at the beginning, then set it to 100% and resume training for 1000s or more, finetuning the text_enc at the end seems to give better results.

I was thinking of doing that but then I never know how many to give to the unet. Right now I did 1500 text then resumed with 3000 unet to see how this goes. In your scenario how do we know when unet is done?

[TheLastBen]

depends on the training subject and the number of instance images, for example, I trained a model on 24 on 30 images, 768px, 7500 steps total with 1000s text_encoder, I got amazing results, it was trained on a photography style

[DarkAlchy]

My experiment was an utter disaster. I trained 1500 of the text encoder and I could even make my model frown. SHOCKED I am. The problem is the face was crap. I do 3k saving from 1500 every 500. All crap even at 1500 where I lost my face and it became someone else. POOF, under trained but had the overtraining pixelated stuff. I wanted to really push it so What I did was train to 3k then to 5k saving every 250 steps. I can say at 4k, and above, it was 100% unusable due to being overtrained. Not using class images is a VERY big deal as on Shiv my models work so beautifully, but his has two issues. It lacks fine-tuning as well as it now takes longer to encode than we get as a free colab. Suddenly, as of yesterday, my 65m became 2h. At 3.5k steps it is 3h and we only get 2.5h per day (which I just spent doing all this testing with TLB).

Having done this much testing I can see a few things and I will turn on the regularization images because it needs them, though they are not the same as I am used to we shall see. I also switched to 3k then screw with the text encoder, but not being local I am so hampered.

Btw, why would an image go B&W then 500s still b&w, and another 500 steps b&w, then 500 more and colour from then on? That shift to b&w while testing was weird since all before, and after, were in colour with each using the exact same settings for uniformity.

Doing your method is weird. I trained the encoder for 1500 steps then trained the unet for 3k via resuming. It just finished and did this for another 1500 steps.

Due to the above I ran out of time on colab and it yanked the GPU. I have nothing to show for this experiment, and having to start all over would end up with the same outcome so can't be done by this free colab user as the 2.5h is just not long enough.

[TheLastBen]

once the text encoder is trained, when you resume, disable "train_text_encoder" to avoid training it again

[DarkAlchy]

once the text encoder is trained, when you resume, disable "train_text_encoder" to avoid training it again

That is the point, I am not stupid, I had it unchecked. If notice it trained for 1500 steps the text_encoder, then the 3k it trained for the unet then it finished and said it was training the unet again but for the 1500 steps I trained the text_encoder for the first round.

Started training today and started with 3200 unet YET it never says it is training unet.

[giteeeeee]

________________Old Method: prior preservation to instance images + prior preservation to text encoder
_______________New Method: prior preservation to instance images + prior preservation to text encoder
Experimental New Method : prior preservation to instance images + prior preservation to text encoder

Am I correct?

[TheLastBen]

Correct 👍

[TheLastBen]

@shyt47 set the total steps to 3000, text encoder to 35%, so the total would be (with the experimental new method) (35% of 3000) text encoder +3000 unet, so around 4500s

edit: sorry, around 4000 not 4500, the pictues above are 1k text_enc, 3k unet

[TheLastBen]

5000 unet 1000 text_enc :

[TheLastBen]

yes

[shyt47]

What is the prompt for above?

[TheLastBen]

_______,still frame by fred herzog, ultra detailed, beautiful, elegant, closeup

negative prompt : cartoon, fake, painting, drawing, 3d, low poly

Steps: 50, Sampler: Euler a, CFG scale: 7, Size: 512x768, Denoising strength: 0.7, First pass size: 0x0, highres.fix

[MasterDenis]

5000 unet 1000 text_enc :

how many images for this one?

[TheLastBen]

20

[giteeeeee]

In the normal new method, say setting 100% text encoder + 3000 training steps, is it just going to train 3000 steps of text encoder and 0 step unet?

But in the experimental new method, if 100% text encoder + 3000 training steps, is it going to train 3000 steps of text encoder AND 3000 steps unet?

If so, how do I make it train only 3000 steps encoder but 0 step unet?

[TheLastBen]

Without training the unet, you won't see any resemblance to the trained subject

[TheLastBen]

in the normal new method if you set 100% text_enc, it will train both text_enc and unet at the same time

with the experimental, it can't train both at the same time because it uses regularization only for the text_enc, so 2 processes are needed for text_enc and unet.

I don't see a reason why you would train only the text_enc without training the unet, for 20 pics, you need only 350 steps of text_enc, adding also 350 to the unet won't hurt

[TheLastBen]

if you set the global step to 3000, 50% text_encoder, it will train 1500 text encoder + 3000 unet

[TheLastBen]

Because it doesn't train the unet during the first 1500 steps, and the unet needs a lot more steps than the text_enc

[giteeeeee]

if you set the global step to 3000, 50% text_encoder, it will train 1500 text encoder + 3000 unet

3000 global steps, 50% text_encoder

Experimental: 1500 encoder, then 3000 unet

Normal: 1500 encoder+1500 unet, then 1500 unet

Is it right?

[TheLastBen]

in normal, when you set 3000, it will train both text_enc and unet for 3000 at the same time

[giteeeeee]

in normal, when you set 3000, it will train both text_enc and unet for 3000 at the same time

In normal, text_enc and unet train at the same time

but when text_enc freezes at 50%/1500 step, then unet is trained alone for the rest of the steps?

[jx1k]

I trained a model yesterday with images of myself. Thirty images, Training_Steps set to 3000 and Contains_faces set to male. The results were comically bad with SD giving me giant lips and an extremely receding chin. I did another run today with Contains_faces set to no and the results have been good so far.
So am I right in assuming, that setting Contains_faces to No just does the training process like it used to be?

[giteeeeee]

Above he told me 5k, 20 images, and 10% I tried it and ugh.
The main problem is getting the model to do as I say, but then the artefacts the higher I push the CFG to get it to do my bidding. I think that is the text encoder. This is why I mentioned to him above (in my edit) we need to stop with the percentage nonsense and allow us to do that or just input our own number irrespective of how many final steps we do.

Based on observation I would say his Method makes it much easier to call my subject, however I have an issue with overfitting, its too easy to overfit compared to the pre-fork version, of course he tries to adjust it accordingly but it seems like its a bit aggressive, it strikes for likeness so far but the issue is that there is a very thin line. So based on my observation its doing well at calling the subject at lower CFG, I was able to get my subject (almost) 1:1 at about below 7 CFG, thats both a good and a bad thing, if you want to prompt-engineer it, it will be difficult to call the subject as 1:1 AND have everything else around it be built in the way you want it without sacrificing something, the weights will be lower on ALL keywords and that could be an issue, ideally I want to have a good-enough likeness at around CFG 7-10, it depends of course. I need to call my subject and have everything else be as it should. Talking from a technical artist's perspective here, that would be "ideal".

So, Shiv's script has an issue with that as well, it makes it virtually impossible to bring the subject below 10 CFG. Also I noticed that we got to be careful how much we train especially with Ben's script.

We have to remember that the users are most likely not coders so those things should be more clear of exactly what they affect and by how much so people can have a better idea of how to configure this.

Currently the 3000k steps 20 images and 31.7% text encoder seemed ideal but when I put more images with the same ratios (so 3500k steps and 31.7% text encoder) that caused a very very bad overfit at CFG 7, I got the most ideal results with: 3000 steps 31.7% text encoder with 23 images (and class images at 175). This was Almost 1:1 (not quiet but close). My goal was to have 1:1 likeness for photo-reproduction. Perhaps also for restoration purposes (which is a bit more challenging considering there are not as many images and we have to improvise), so 1:1 is important in those use cases.

Have you experimented with the new method using 20 images? I've been using 30 images which might be too many?

[DarkAlchy]

[snip]

Have you experimented with the new method using 20 images? I've been using 30 images which might be too many?

I did 20 images, 5k steps (saved every 1k steps) and 10% text encoding per Ben. Not good.

[TheLastBen]

You can never find the best settings using instance images taken with average hardware, download high quality pictures from google of some known person, crop them with a good software and find the best settings, then apply those to your own instance pics. 99% of the problems are caused by the input pictures

[DarkAlchy]

Here is a pic of the training just calling it out using a good seed using the new DPM++2S Karras at 20 steps and CFG 7.

I caught a lucky seed but if I try and use the model to do something it begins to fall apart. Either it ignores the request and looks alright or it does it and the artefacts/noise in the background is there. Oh, let me add that is using the 2k step save as more than that it begins to look ugly no matter. The 4k/5k is pretty bad.

[d1mski]

I have come to the same conclusion. So far what I find to work well is using around 20-22 images, and training the text_enc for 30%. So in general as long as the unet is trained for approximately 2.5 times the steps used for the text_enc, things are working fairly well and I'm able to stylise my models with CFG set to 7. I had badly trained models but solely depending on input images.

Not using contains_faces and on standard GPU (I believe it's a T4). Your INPUT images have to be of good quality in good light conditions.

P.S. I will try to train the text_enc in the end as you suggested Ben and see if I get better results.

[pcaben]

I have a "interesting" experience with the contain face feature, I am a asian guy have long hair but most of the photo I use for training is me with ponytail hair, so should be not super outstanding.

I choose men in contain face feature expected better training for men. In the final result that fine to create photo of me and the dress style just like the photo I use. But if I add more prompts for create painting e.g (fantasy, intricate, highly detailed, digital painting, artstation, concept art, smooth, sharp focus, illustration, art by artgerm and greg rutkowski and alphonse mucha). Something you standard painting prompts you can find in google.

The result will make me being a woman with breasts, I need to add extra prompts (man) to make myself be a man in the painting but also change my face a bit. lol...

[bparrott78]

I re-ran the same pictures I had done with the old method through this new method and I can't say I really like it. I tested the model every 500 steps and it just never spat out an image that was the likeness of the person being trained. I had 20 face shots and 6 body shots from the front, back and sides. With some VERY specific parameters and negatives I was able to get some decent results. I have to say that the pictures themselves are good, but the face really doesn't match.

[TheLastBen]

@bparrott78 set the images resolution to 640 and train at 640, make sure the quality of the pictures is high, not a phone quality

[bparrott78]

Tried this, same result. Maybe I'm using too many pictures?
High quality pictures of beautiful women who look similar but definitely not the same.

[bparrott78]

Tested again, this time I ran the Shivam shirao notebook as well. Used the exact same images, saved every 500 to compare, trained on pictures of my wife.
My results were this:
Shivam with prior preservation on at 2500 steps was the best representation of her.
While the images that came from thelastben notebook were higher quality, the face seemed like it had been merged with other women.
Going back and comparing against previous models I had trained, this was not the case. This new method may be good for styles but I can't seem to get a good representation of a specific person.

[TheLastBen]

set the text encoder to 100%