fix 3 xpu failures uts w/ latest pytorch#12408

DN6 merged 1 commit intohuggingface:mainhuggingface/diffusers:mainfrom

yao-matrix:issue-531yao-matrix/diffusers:issue-531Copy head branch name to clipboard