Add support for XFormers in SD3

[CanvaChen]

What does this PR do?

Add support for XFormers in SD3 (#8535)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sayakpaul @yiyixuxu

[CanvaChen]

Hi @yiyixuxu @sayakpaul , could you please take a moment to review this PR when you have time? Thank you!

[yiyixuxu]

thanks!

[sayakpaul]

Thanks!

Could you also provide a test script for this so that we can run some tests?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[CanvaChen]

def test_sd3(model_path, prompt):
    pipe = StableDiffusion3Pipeline.from_pretrained(model_path, torch_dtype=torch.float16).to('cuda')
    pipe.enable_xformers_memory_efficient_attention()

    image = pipe(prompt, negative_prompt='',
                 num_inference_steps=28, guidance_scale=7.5,
                 height=1024, width=768).images[0]

[sayakpaul]

@DN6 how should this test set be added?

[CanvaChen]

Do you mean adding a test script to the 'tests' folder for submission?

[DN6]

Pipeline tests for xformers is handled via the PipelineTesterMixin, so an additional pipeline test is not needed. It would be nice to add a test in for the SD3 Transformer though. Similar to how we have it here

diffusers/tests/models/unets/test_models_unet_2d_condition.py

Line 398 in c1dc2ae

def test_xformers_enable_works(self):

[CanvaChen]

============================= test session starts ==============================
collecting ... collected 1 item
models/transformers/test_models_transformer_sd3.py::SD3TransformerTests::test_xformers_enable_works PASSED [100%]
============================== 1 passed in 2.16s ===============================

done @sayakpaul @DN6

[DN6]

@CanvaChen Could you please run make style && make quality for the QC checks to pass?

[CanvaChen]

done @DN6 @sayakpaul

ruff check examples scripts src tests utils benchmarks setup.py
ruff format --check examples scripts src tests utils benchmarks setup.py
1069 files left unchanged
doc-builder style src/diffusers docs/source --max_len 119 --check_only
python utils/check_doc_toc.py

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sayakpaul]

Apologies for this delay. @DN6 could you give this a review?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sayakpaul]

@CanvaChen I think this is still quite relevant. Would you maybe like to resolve the conflicts so that we can attempt a merge?

[CanvaChen]

@sayakpaul @DN6 I have resolved the code conflicts and also incorporated the recent changes from JointAttnProcessor2_0.

[sayakpaul]

@DN6 could you please give this a review?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sayakpaul]

Failing tests are unrelated.