LORA resuming from existing safetensors gives visual noise

[haltil]

I'm using the sd-scripts from the sd3 branch to train for Flux 1.D. I can train a LORA from my dataset from scratch and it works pretty well. I can also resume training the first LORA after it's finished using the "load network weights" arg with no problems. I can also apply a second LORA I downloaded together with my trained one at inference and it works.

As an experiment I tried instead starting my training from the downloaded LORA using "load network weights". The sample images during training look perfect. But every image generated by the final LORA is just visual noise like this. Even the very first save at the beginning of training gives this output.

I guess there's something different about the contents of the downloaded LORA that doesn't stop kohya being able to use it at inference or to train on, but stops it saving as a valid safetensors. I noticed the show_metadata.py script says "no metadata" when I run it on the downloaded file. Not sure if that's relevant. Any ideas what is happening or how to fix it?

I think it may be related to quantization because the images look very similar to these: huggingface/optimum-quanto#343