No learning during training with xformers 0.0.16rc425

[kernalkue]

Is anybody else having issues with training producing no learning with the new xformers? I have no errors, just couldn't get my embeddings or DB to learn regardless of how high I set the learning rate.

Eventually this morning I tried without xformers and they started learning again. Is this just me?

[vladmandic]

see #6746

[kernalkue]

Sorry, I'm not seeing the connection between the two

[vladmandic]

if you're using xformers make sure that settings Use cross attention optimizations while training is disabled. that basically removes xformers cross-optimization at the start of the training and re-applies it at the end of training. training with xformers enabled does not do anything much as results are frozen.

[kernalkue]

Awesome. I'll check that out

[Zuxier]

it's an issue with every xformers after 0.14dev0. Plus cu117 has worst training.

[kernalkue]

To be clear everything looks (on the face of it) like it is working, but my samples never take any aspects of my subject.

[Wannafly7]

Yesterday I ran with xformers and everything worked, but now it doesn't work. Pictures do not change, or change very little, and then reset again to the beginning.

[Wannafly7]

Use cross attention optimizations while training is off