[Tests] Enable more general testing for torch.compile() with LoRA hotswapping

[sayakpaul]

What does this PR do?

@BenjaminBossan shipped a great feature in #9453. We decided to add tests only for the Unet2DConditionModel. However, we support LoRA hotswapping with torch.compile() for all models subclassed with PeftAdapterMixin.

This PR refactors the TestLoraHotSwappingForModel class to make it more generally available to the model test classes. I believe this is nice to have as this feature is for power users and for users who use diffusers in production environments.

Currently, this PR adds LoraHotSwappingForModelTesterMixin (revised name) to the test classes of Unet2DConditionModel and FluxTransformer2DModel as I think those are the most popular ones having significant usage.

I think we can open the rest of the models for community contributions.

@yiyixuxu FYI.

[BenjaminBossan]

Thanks for refactoring the tests to make them modular and align them better with the diffusers testing style. I don't have anything to add, just a few nits. I was not sure if the tests are run in the CI so I ran them locally and they all passed (they took 2 min 7 sec).

[sayakpaul]

I was not sure if the tests are run in the CI so I ran them locally and they all passed (they took 2 min 7 sec).

Just added it to our nightly CI: 4b11ab2.

[DN6]

Just a question. Why not place these under the LoRA tests in the nightlys? The compile image isn't actually needed anymore. It was only a workaround for issues we were facing with Python 3.8 and torch.compile. Since the base images now using 3.10, we should be okay to have them run there.

[sayakpaul]

@DN6 good point. Done.

[sayakpaul]

@DN6 a gentle ping.