[LoRA] refactor lora loading at the model-level#11719

sayakpaul merged 14 commits intomainhuggingface/diffusers:mainfrom

load-lora-method-refactorhuggingface/diffusers:load-lora-method-refactorCopy head branch name to clipboard