Fail to merge for a custom model.

[Boltzmachine]

I have a custom model. I use peft's lora to train it and it is automatically saved by huggingface's trainer. When I load it by AutoModel.from_pretrained, I found the type of the model is the custom model (not any of the peft classes), yet the linear layer is replaced by lora.linear. However, using such model is slow when inference so I want to the lora weights. I tried following methods

1. model.merge_and_unload(). Failed because the model is not a peft model when I load it. It is the type of my custom model instead so it does not have such a method.
2. AutoPeftModelForCausalLM.from_pretrained(path). Failed with error message:

Unrecognized configuration class <class 'transformers_modules.spatialvla-4b-224-pt.configuration_spatialvla.SpatialVLAConfig'> for this kind of AutoModel: AutoModelForCausalLM.
Model type should be one of BartConfig ...

3. model = AutoModelForCausalLM(original_path_of_the_base_model); model.load_adapter(...). Failed, it is the same as 1.

[BenjaminBossan]

Could you please provide a bit more code so that we can better understand what's going on here? No need for the full training code, but how you initialize the model before training, how you save it, how you load it, and how you try to merge it.

[Boltzmachine]

1. Training: CustomModelForConditionalGeneration.from_pretrained
2. Save: I just use a huggingface trainer and it saves the checkpoint internally.
3. load: I used AutoModel.from_pretrained(checkpoint_path). And got a model of type CustomModelForConditionalGeneration but with lora.Linear injected.
4. Merge: I cannot merge it since it is not a peft model. If I load it with AutoPeftModelForCausalLM.from_pretrained, I will get the error in point 2 of my initial post.

[BenjaminBossan]

Could you try the following (steps 1 and 2 stay the same):

3. Load base model as usual: model = CustomModelForConditionalGeneration.from_pretrained(...).
4. Load LoRA adapter: model = PeftModel.from_pretrained(model, checkpoint_path).
5. Then merge: model = model.merge_and_unload().