[Core] Set `weights_only=True` nicely when we use `torch.load()`.#9455

sayakpaul wants to merge 5 commits intomainhuggingface/diffusers:mainfrom

torch-load-weights-onlyhuggingface/diffusers:torch-load-weights-onlyCopy head branch name to clipboard