enable lora cases on XPU#11506

a-r-r-o-w merged 3 commits intohuggingface:mainhuggingface/diffusers:mainfrom

yao-matrix:lora-xpuyao-matrix/diffusers:lora-xpuCopy head branch name to clipboard