Remove device synchronization when loading weights#11927

a-r-r-o-w merged 2 commits intomainhuggingface/diffusers:mainfrom

fix-remove-sync-devicemaphuggingface/diffusers:fix-remove-sync-devicemapCopy head branch name to clipboard