[CI] Speed up GPU PR Tests#11887

DN6 merged 2 commits intomainhuggingface/diffusers:mainfrom

gpu-pr-testhuggingface/diffusers:gpu-pr-testCopy head branch name to clipboard