[LoRA] fix: disabling hooks when loading loras.#11896

sayakpaul merged 4 commits intomainhuggingface/diffusers:mainfrom

fix-disabling-hookshuggingface/diffusers:fix-disabling-hooksCopy head branch name to clipboard