Flux Can't Load LoRA

[markuryy]

I'm getting this error attempting to load lora for flux trained with SimpleTuner:
AttributeError: 'FluxPipeline' object has no attribute 'load_adapter'
Desperate to get this working

import torch
from diffusers import DiffusionPipeline

# Load the pre-trained model and adapter
model_id = 'black-forest-labs/FLUX.1-dev'
adapter_id = 'markury/AndroFlow-FLUX-SimpleTuner-1'
pipeline = DiffusionPipeline.from_pretrained(model_id)
pipeline.load_adapter(adapter_id)

AttributeError                            Traceback (most recent call last)
[<ipython-input-5-51e24f338356>](https://localhost:8080/#) in <cell line: 8>()
      6 adapter_id = 'markury/AndroFlow-FLUX-SimpleTuner-1'
      7 pipeline = DiffusionPipeline.from_pretrained(model_id)
----> 8 pipeline.load_adapter(adapter_id)

[/usr/local/lib/python3.10/dist-packages/diffusers/configuration_utils.py](https://localhost:8080/#) in __getattr__(self, name)
    141             return self._internal_dict[name]
    142 
--> 143         raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
    144 
    145     def save_config(self, save_directory: Union[str, os.PathLike], push_to_hub: bool = False, **kwargs):

AttributeError: 'FluxPipeline' object has no attribute 'load_adapter'

[markuryy]

I'm using the lora-support-flux branch
!pip install git+https://github.com/huggingface/diffusers@lora-support-flux

[bghira]

there is a messy workaround. you can use the same style of loading that we do inside simpletuner.

from diffusers.loaders import LoraLoaderMixin
from diffusers.utils import (
    convert_state_dict_to_diffusers,
    convert_unet_state_dict_to_peft,
)
from peft import set_peft_model_state_dict
from peft.utils import get_peft_model_state_dict

lora_state_dict = FluxPipeline.lora_state_dict('/path/to/lora/weights')
transformer_state_dict = {
    f'{k.replace("transformer.", "")}': v
    for k, v in lora_state_dict.items()
    if k.startswith("unet.")
}
transformer_state_dict = convert_unet_state_dict_to_peft(
    transformer_state_dict
)
incompatible_keys = set_peft_model_state_dict(
    transformer_, transformer_state_dict, adapter_name="default"
)
if incompatible_keys is not None:
    # check only for unexpected keys
    unexpected_keys = getattr(incompatible_keys, "unexpected_keys", None)
    if unexpected_keys:
        print(
            f"Loading adapter weights from state_dict led to unexpected keys not found in the model: "
            f" {unexpected_keys}. "
        )

obviously it's not in a state where it'll work, that's just the relevant bits and pieces pulled from the save_hooks module

but maybe it'll help

[bghira]

@sayakpaul @yiyixuxu @asomoza if you need a lora for testing: https://huggingface.co/mikaelh/flux-sanna-marin-lora-v0.1/blob/main/checkpoint-1950/pytorch_lora_weights.safetensors

[markuryy]

It should be pipeline.load_lora_weights(adapter_id) instead of pipeline.load_adapter(adapter_id).

[bghira]

no there is still a bug if load_adapter doesn't work, as this is the PEFT method, and load_lora_weights is the old style

[sayakpaul]

Nope. load_lora_weights() is NOT the old style:
https://huggingface.co/docs/diffusers/main/en/tutorials/using_peft_for_inference

In fact there is no method called load_adapter() at the pipeline level:
https://github.com/search?q=repo%3Ahuggingface%2Fdiffusers%20load_adapter&type=code

[bghira]

well, i stand corrected :P it has had so many ways of doing it. thanks for the links.

[sayakpaul]

@markuryy okay to close?

[Kathy-Peng]

tried load_lora_weights function but the images generated look identical for lora and base. Don't think it's applied correctly.

[sayakpaul]

Check out #9134.

[kishlaykumar1995]

I tried to load_lora_weights for a model I trained on CivitAI but it doesn't load. But somehow the InferenceAPI on huggingface hub is able to load it and generate images.

[sayakpaul]

Do you have a reproducible code snippet?