Textual Inversion giving lego clowns from hell. Any ideas?

[SorenTruelsen]

Hi Ben and the team
Been trying to use textual inversion for a couple of days (around 100 colab-units). All my embeddings give "Yellow Lego Clown Figures".

I have done loads of different variations to try to find a mistake in my setup; used different learning rates, initialization text, number of vectors, picture sets, prompts, training steps. At around 25K they solidify in this ghastly horror expression.

The setting in GUI for the attached image was:
Initialization text: Woman posing secretary
Vectors: 3
Learning rate: 0.005
Batch size: 1
Gradient accumulation steps: 1
Choose latent sampling method: Once

Not sure if its a bug, but I really haven't been able to locate anything to explain this artifact. A suggestion from Discord was it was related to CFG setting or Sampler, but since the GUI is not really offering to change this (other than for the preview image), I am not sure how to play around with that.

Thanks

[TheLastBen]

25k steps is too much, even for textual inversion, have you tried lower steps and lower learning rate ?

[SorenTruelsen]

The 30.000 were my stating points.

The images shown here are from 500, 1000, 1500 and 2000 steps. It only got worse from then on. Except the concept got even more clear

The learning rate, I have tried to reduce to 0.0005 (a tenth of the the default). Will try again now with a new set of images.

Running on colab pro+, and have tried the different GPU-settings to see if that was the reason. Also, tried setting captions manual instead of using BLIP/Deepbooru