extract lora deltas from models

[adrianpuiu]

would anyone be kind enough to implement the lora extraction script from checkpoints
https://github.com/bmaltais/kohya_ss/blob/master/networks/extract_lora_from_models.py into automatic1111 ?

this way we end up with the base model and plenty of lora files and save some storage ...

[f-rank]

@adrianpuiu
does this comment happen to be related to your request ?

https://civitai.com/models/6150/alicia-vikander?commentId=17306&modal=commentThread

[adrianpuiu]

Hi F-rank, I don't know ... could be something similar ... I might be wrong... I just know i've used the kohya tools to extract the Lora model from couple of checkpoints

[toyxyz]

You can use this script to extract the difference between two models with Lora. but it's not perfect https://github.com/bmaltais/kohya_ss/blob/master/train_network_README.md

[adrianpuiu]

python networks\extract_lora_from_models.py --model_org base-model.ckpt
--model_tuned fine-tuned-model.ckpt
--save_to lora-weights.safetensors --dim 4 . this is what i've used before ... but was wondering if it would be possible to have it included in the wui1111 backend/frontend ..

[FartyPants]

Does it work though? Is the Lora usable this way?

[toyxyz]

Does it work though? Is the Lora usable this way?
Unfortunately, the extracted Lora is not perfect.
At least in my tests it is.

Original / Dreambooth / Lora / Lora extracted from Dreambooth

[adrianpuiu]

Does it work though? Is the Lora usable this way?

yes it works, it affects the output as it should.

[FartyPants]

That's good to know. I tried the gui with some of my checkpoints and got a questionable results, but I'll try a bit more.
Although for anybody interested, the link to the GUI: https://github.com/bmaltais/kohya_ss
is the easy way to go IMHO It's venv so it will no clash with Auto. Not sure how integrating this will be better than what there already is - but who knows.

[fractal-fumbler]

hello :) anybody had success in extracting lora weights from SD-2.X models and using them with webui?

with fp16 weights i am getting wall of keys

failed to match keys when loading Lora

ailed to match keys when loading Lora /tmp/stable-diffusion-webui/models/Lora/dblx_320.safetensors: ['lora_te_text_model
_encoder_layers_0_mlp_fc1.alpha', 'lora_te_text_model_encoder_layers_0_mlp_fc1.lora_down.weight', 'lora_te_text_model_en
coder_layers_0_mlp_fc1.lora_up.weight', 'lora_te_text_model_encoder_layers_0_mlp_fc2.alpha', 'lora_te_text_model_encoder
_layers_0_mlp_fc2.lora_down.weight', 'lora_te_text_model_encoder_layers_0_mlp_fc2.lora_up.weight', 'lora_te_text_model_e
ncoder_layers_0_self_attn_k_proj.alpha', 'lora_te_text_model_encoder_layers_0_self_attn_k_proj.lora_down.weight', 'lora_
te_text_model_encoder_layers_0_self_attn_k_proj.lora_up.weight', 'lora_te_text_model_encoder_layers_0_self_attn_out_proj
.alpha', 'lora_te_text_model_encoder_layers_0_self_attn_out_proj.lora_down.weight', 'lora_te_text_model_encoder_layers_0
_self_attn_out_proj.lora_up.weight', 'lora_te_text_model_encoder_layers_0_self_attn_q_proj.alpha', 'lora_te_text_model_e
ncoder_layers_0_self_attn_q_proj.lora_down.weight', 'lora_te_text_model_encoder_layers_0_self_attn_q_proj.lora_up.weight
', 'lora_te_text_model_encoder_layers_0_self_attn_v_proj.alpha', 'lora_te_text_model_encoder_layers_0_self_attn_v_proj.l
ora_down.weight', 'lora_te_text_model_encoder_layers_0_self_attn_v_proj.lora_up.weight', 'lora_te_text_model_encoder_lay
ers_10_mlp_fc1.alpha', 'lora_te_text_model_encoder_layers_10_mlp_fc1.lora_down.weight', 'lora_te_text_model_encoder_laye
rs_10_mlp_fc1.lora_up.weight', 'lora_te_text_model_encoder_layers_10_mlp_fc2.alpha', 'lora_te_text_model_encoder_layers_
10_mlp_fc2.lora_down.weight', 'lora_te_text_model_encoder_layers_10_mlp_fc2.lora_up.weight', 'lora_te_text_model_encoder
_layers_10_self_attn_k_proj.alpha', 'lora_te_text_model_encoder_layers_10_self_attn_k_proj.lora_down.weight', 'lora_te_t
ext_model_encoder_layers_10_self_attn_k_proj.lora_up.weight', 'lora_te_text_model_encoder_layers_10_self_attn_out_proj.a
lpha', 'lora_te_text_model_encoder_layers_10_self_attn_out_proj.lora_down.weight', 'lora_te_text_model_encoder_layers_10
_self_attn_out_proj.lora_up.weight', 'lora_te_text_model_encoder_layers_10_self_attn_q_proj.alpha', 'lora_te_text_model_
encoder_layers_10_self_attn_q_proj.lora_down.weight', 'lora_te_text_model_encoder_layers_10_self_attn_q_proj.lora_up.wei
ght', 'lora_te_text_model_encoder_layers_10_self_attn_v_proj.alpha', 'lora_te_text_model_encoder_layers_10_self_attn_v_p
roj.lora_down.weight', 'lora_te_text_model_encoder_layers_10_self_attn_v_proj.lora_up.weight', 'lora_te_text_model_encod
er_layers_11_mlp_fc1.alpha', 'lora_te_text_model_encoder_layers_11_mlp_fc1.lora_down.weight', 'lora_te_text_model_encode
r_layers_11_mlp_fc1.lora_up.weight', 'lora_te_text_model_encoder_layers_11_mlp_fc2.alpha', 'lora_te_text_model_encoder_l
ayers_11_mlp_fc2.lora_down.weight', 'lora_te_text_model_encoder_layers_11_mlp_fc2.lora_up.weight', 'lora_te_text_model_e
ncoder_layers_11_self_attn_k_proj.alpha', 'lora_te_text_model_encoder_layers_11_self_attn_k_proj.lora_down.

float weights giving error too

RuntimeError: output with shape [320, 320] doesn't match the broadcast shape [320, 320, 320, 320]

tivating extra network lora with arguments [<modules.extra_networks.ExtraNetworkParams object at 0x7f7b41cbca00>]: F neError
Traceback (most recent call last):
File "/tmp/stable-diffusion-webui/modules/extra_networks.py", line 75, in activate extra_network.activate(p, extra_network_args)
File "/tmp/stable-diffusion-webui/extensions-builtin/Lora/extra_networks_lora.py", line 23, in activate
lora. load_loras (names, multipliers)
File "/tmp/stable-diffusion-webui/extensions-builtin/Lora/lora.py", line 151, in load_loras Lora = load_lora (name, lora_on_disk. filename)
File "/tmp/stable-diffusion-webui/extensions-builtin/Lora/lora.py", line 113, in load_lora module.weight.copy_(weight)
RuntimeError: output with shape [320, 320] doesn't match the broadcast shape [320, 320, 320, 320]