[chore] remove extra validation check in determine_device_map#12176

sayakpaul merged 3 commits intomainhuggingface/diffusers:mainfrom

remove-extra-validationhuggingface/diffusers:remove-extra-validationCopy head branch name to clipboard