[tests] Refactor TorchAO serialization fast tests#10271

DN6 merged 5 commits intomainhuggingface/diffusers:mainfrom

refactor-torchao-serialization-testshuggingface/diffusers:refactor-torchao-serialization-testsCopy head branch name to clipboard