[tests] fix AssertionError: Torch not compiled with CUDA enabled

[faaany]

What does this PR do?

The following test cases return AssertionError: Torch not compiled with CUDA enabled on XPU. This PR fixes this issue.

pytest -rA tests/single_file/test_stable_diffusion_xl_adapter_single_file.py::StableDiffusionXLAdapterPipelineSingleFileSlowTests::test_single_file_format_inference_is_same_as_pretrained
pytest -rA tests/single_file/test_stable_diffusion_xl_controlnet_single_file.py::StableDiffusionXLControlNetPipelineSingleFileSlowTests::test_single_file_format_inference_is_same_as_pretrained
pytest -rA tests/single_file/test_stable_diffusion_xl_img2img_single_file.py::StableDiffusionXLImg2ImgPipelineSingleFileSlowTests::test_single_file_format_inference_is_same_as_pretrained
pytest -rA tests/single_file/test_stable_diffusion_xl_img2img_single_file.py::StableDiffusionXLImg2ImgRefinerPipelineSingleFileSlowTests::test_single_file_format_inference_is_same_as_pretrained
pytest -rA tests/single_file/test_stable_diffusion_xl_single_file.py::StableDiffusionXLPipelineSingleFileSlowTests::test_single_file_format_inference_is_same_as_pretrained

[hlky]

Thank you @faaany