Please restore the fp16 tickbox

[SAC020]

Hi,

Just realized I need to throw away a session because it got trained in fp16 and I haven't noticed the tickbox is gone (I am aware I can change the option in code, but haven't realized I need to do so because the option is gone).

Please bring it back

TYVM

[TheLastBen]

Why throw it away ? The results are exactly the same between fp16 and fp32

[SAC020]

Why throw it away ? The results are exactly the same between fp16 and fp32

Not if I want to retrain it further (as far as I know)

[TheLastBen]

You can retrain it as much as you want, and if you are using the A100 premium GPU, fp32 is activated automatically (the CKPTs are > 4GB)

[SAC020]

It is possible to train 768px on the standard colab GPU, it does not run out of VRAM. I'm using standard GPU always.

I hear what you say about fp16 being re-trainable, I feel better to have fp32 considering all the complexity and variables around training

[TheLastBen]

fp32 cuts the speed by half and doesn't have any effect on the output, it's just a placebo effect

[SAC020]

I've tried to train an embedding on a SD 2.1 fp16 ckpt and I am getting "nan".

On a different SD 2.1 fp32 model I get normal loss readouts. Is embedding training possible on fp16? If not, how do I generate a 4GB ckpt out of a 2GB one?

[TheLastBen]

if you really want a 4gb ckpt, change "fp16 = True" to "fp16 = False" in the training cell

[SAC020]

I am well aware of that.

My questions are:

• to your knowledge, is embedding training possible on 2GB?
• how can I convert a 2GB to a 4GB? Load the 2GB, train 1 step, set "fp16 = false"? would that work?

[TheLastBen]

yes that would work, as for the embeddings, they should work with any model

[SAC020]

This is training on a 4GB ckpt

This is on 2GB (Loss = nan <=> embedding is screwed)

I will try the above workaround when I finish the current batch

[SAC020]

"conversion" to 4GB works, unfortunately the embedding training returns "nan" on the "converted" model too

[SAC020]

I apologize, I was mistaken (although the whole 2.0/2.1 environment is quite confusing at the moment)

I don't know about training embeddings on 2GB, haven't tested yet on a SD1.5 model, however the embedding issue above is not caused by the 2GB checkpoint, but by a training issue with SD2.1 overall.

More info here: AUTOMATIC1111/stable-diffusion-webui#5715

I will be dropping SD20/21 for a while until it stabilizes, too many bugs and issues atm

[TheLastBen]

Getting the same nan output, I'm trying a new method as soon as textual inversion is fixed for V2.1