make group offloading work with disk/nvme transfers#11682

sayakpaul merged 19 commits intomainhuggingface/diffusers:mainfrom

group-offloading-with-diskhuggingface/diffusers:group-offloading-with-diskCopy head branch name to clipboard