[Flux] reduce explicit device transfers and typecasting in flux.#9817

sayakpaul merged 5 commits intomainhuggingface/diffusers:mainfrom

reduce-explicit-device-transfershuggingface/diffusers:reduce-explicit-device-transfersCopy head branch name to clipboard