Noob text encoder percentage question

[spnylbstr]

I'm probably overthinking this, but can anyone confirm whether or not the text encoder percentage is a total percentage? That is, if I want to fine-tune the text encoder percentages over several training sessions, do I need to adjust each run's percentage in accordance with the totals I'll end up with?

For example, assume I run a training set for 5,000 steps with 40% text encoder training. I then want to train for an additional 3,000 steps while increasing the text encoder percentage to a total of 50%. Am I correct in understanding that, in this particular case, I would need to set the second run for ~67% text encoder training in order to achieve the desired result?
That is:

Session 1: 5,000 training steps, 40% text encoder (2,000 text encoder steps)
Session 2: 3,000 training steps, ~67% text encoder (~2,000 text encoder steps)
Total result: 8,000 training steps, ~50% text encoder (~4,000 text encoder steps)

Similarly, I assume that a single-session run of 8,000 training steps and 50% text encoder steps would produce (approximately) the same results, though based on other discussions I've seen, I assume some variance would result.

As I said, I'm probably badly overthinking this, but if someone could confirm whether or not I'm understanding it correctly, I'd appreciate it! Thanks in advance!

[TheLastBen]

If you train the text encoder for a total 800 up to 1000 steps, you simply stop training it and disable "train text encoder" for the rest of the training runs.

[spnylbstr]

I'm less concerned about overfitting (in this case) and more about understanding how that percentage is calculated.
In the example you provided above, I'd interpret that as either 12% (1200 + 0 + 0 = 1200 steps) or 5% (150 + 150 + 200 = 500 steps) since it's out of 10,000 steps total. Is that the correct way to interpret what's going on?

I'm trying to figure it out because I want to compare the results of consistent percent values (e.g., 50% of N total steps, with N varying) to consistent step values (e.g., 5000 steps out of N total steps) and so forth, but I'm not sure if I'm understanding what the percentage is exactly.

[TheLastBen]

it's simple how the percentage is calculated, it's simply Training_Steps*Train_text_encoder_for/100
if you resume training, you should take in account the previous text_enc training and decide whether to train it more or not.

[spnylbstr]

Okay, so just to confirm, in a case where I trained it with two sessions, one for N steps at X% text encoder training and one for Q steps at Y% text encoder training, the total text encoder percentage after training both sessions would be:

100*(N+Q)/(N*(X/100)+Q*(Y/100)) = total text encoder in %

Is that correct?

[TheLastBen]

Yes but the I used % instead of number of steps to make things easy, not to take the % as a rule to follow, in general, the text_encoder should not be finetuned above 1000 steps, no matter the total steps or the number of instance images, just make sure the total of tex_encoder steps is kept low.

[spnylbstr]

Okay, that clears everything up, and makes much more sense. I'm definitely going to try some new runs in that case, because my encoder step counts are all between 1,280 and 4,000+, so it'll be really interesting to see the changes I get with <1000 stepcount numbers!

Thanks so much!

[Filarh]

About Text Encoder, i need to know how Intermediary checkpoints works, idk if text enc is added when training is completed, cuz this image is 3500_steps checkpoint

405 steps at 100%, resuming for 3745 at 7%, V2-768

and the other one is the finished 4200 one.

I mean, I don't think there's that much of a difference between 245 steps, hence my uncertainty. same settings and seed for both

commenting here cuz i don't wanna bother dev opening an issue

[TheLastBen]

the intermediary checkpoint is saved the exact way the final checkpoint is saved, 200 training steps can make a difference at inference