Revert "[tests] Changes to the `torch.compile()` CI and tests"#11613

DN6 wants to merge 1 commit intomainhuggingface/diffusers:mainfrom

revert-11508-compile-cihuggingface/diffusers:revert-11508-compile-ciCopy head branch name to clipboard