Textual inversion - textual_inversion_loss.csv

[adunato]

Do we all agree that there is no use of textual_inversion_loss.csv from a user perspective?

I understand that training loss does not have a direct relationship with training quality, but I have never seen a graph for textual inversion "converging", all I have seen is the same fluctuating moving average which never trends down consistently, even when the "subjective" quality fo the training is good.

I would love to be proven wrong, and see that someone is getting some use from an objective measure, rather than just eyeballing whether a textual inversion training session is good or not.

[zrichz]

I fully agree that the loss is basically a sanity-check for most embeddings, although for significantly larger batch sizes and low learning rates, there is a visible downward trend (if all goes well), which is reassuring

[morphinapg]

That's interesting. I was under the impression that you'd want a higher learning rate for a higher batch size. Do you have some tips?

[morphinapg]

I tried using a batch size of 24 with my normal learning rate schedule compressed into about 5x less steps, and it still looks just as noisy to me in the csv with no visible trend:

The only real difference is that the range of points in the graph are closer together, but no real downward trend that I can see. This is 6000 steps out of a planned 20,000, but I doubt the rest of the steps will make much of a difference in the graph.

That said, the results are MUCH better. Like, the resulting embedding at 6000 steps here with 1 vector is better than I was getting with 100,000 steps with either 1 or 2 vectors. It is incredibly impressive how much better it looks. It will take longer to complete training the full number of steps I planned, and use more power and make my room hotter lol, but it might be worth it, at least for subjects that aren't coming out great with the default batch size of 1.

[zrichz]

sorry i meant larger batches OR low LR's - probably not what we'd get to with embeddings anyway - my bad. Increasing the batch size (or indeed the Gradient Acculumlation) will almost certainly help training converge more smoothly onto the target - effectively you are moving away from lots of pretty random movements to fewer, more accurate movements (as long as your LR is appropriate). congrats on a good embedding! - vector-wise I see ppl releasing embeddings of single people with 12 or 16 vectors and think ouch that's not gonna be flexible - I too use 1 or 2 vectors in my embeddings and they work fine for me (amazing really, as it's just 784 numbers!)

[ghost]

Good morning Thanks for your reply on this video is my face Thank u again i apreciat if u pull out all my video

…

On Tue, 21 Feb 2023, 04:26 morphinapg, ***@***.***> wrote: That's interesting. I was under the impression that you'd want a higher learning rate for a higher batch size. Do you have some tips? — Reply to this email directly, view it on GitHub <#7777 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A5ETC3P5TL2KRELXWHIC3Y3WYQ7ZFANCNFSM6AAAAAAUZRBHYM> . You are receiving this because you are subscribed to this thread.Message ID: <AUTOMATIC1111/stable-diffusion-webui/repo-discussions/7777/comments/5059728 @github.com>