Skip to content

Commit b79e720

Browse files
committed
Merge branch 'main' into variants-fetching-fix
2 parents 3db5a69 + 454f82e commit b79e720

File tree

325 files changed

+21919
-3928
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

325 files changed

+21919
-3928
lines changed

.github/workflows/pr_style_bot.yml

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
name: PR Style Bot
2+
3+
on:
4+
issue_comment:
5+
types: [created]
6+
7+
permissions:
8+
contents: write
9+
pull-requests: write
10+
11+
jobs:
12+
run-style-bot:
13+
if: >
14+
contains(github.event.comment.body, '@bot /style') &&
15+
github.event.issue.pull_request != null
16+
runs-on: ubuntu-latest
17+
18+
steps:
19+
- name: Extract PR details
20+
id: pr_info
21+
uses: actions/github-script@v6
22+
with:
23+
script: |
24+
const prNumber = context.payload.issue.number;
25+
const { data: pr } = await github.rest.pulls.get({
26+
owner: context.repo.owner,
27+
repo: context.repo.repo,
28+
pull_number: prNumber
29+
});
30+
31+
// We capture both the branch ref and the "full_name" of the head repo
32+
// so that we can check out the correct repository & branch (including forks).
33+
core.setOutput("prNumber", prNumber);
34+
core.setOutput("headRef", pr.head.ref);
35+
core.setOutput("headRepoFullName", pr.head.repo.full_name);
36+
37+
- name: Check out PR branch
38+
uses: actions/checkout@v3
39+
env:
40+
HEADREPOFULLNAME: ${{ steps.pr_info.outputs.headRepoFullName }}
41+
HEADREF: ${{ steps.pr_info.outputs.headRef }}
42+
with:
43+
# Instead of checking out the base repo, use the contributor's repo name
44+
repository: ${{ env.HEADREPOFULLNAME }}
45+
ref: ${{ env.HEADREF }}
46+
# You may need fetch-depth: 0 for being able to push
47+
fetch-depth: 0
48+
token: ${{ secrets.GITHUB_TOKEN }}
49+
50+
- name: Debug
51+
env:
52+
HEADREPOFULLNAME: ${{ steps.pr_info.outputs.headRepoFullName }}
53+
HEADREF: ${{ steps.pr_info.outputs.headRef }}
54+
PRNUMBER: ${{ steps.pr_info.outputs.prNumber }}
55+
run: |
56+
echo "PR number: ${{ env.PRNUMBER }}"
57+
echo "Head Ref: ${{ env.HEADREF }}"
58+
echo "Head Repo Full Name: ${{ env.HEADREPOFULLNAME }}"
59+
60+
- name: Set up Python
61+
uses: actions/setup-python@v4
62+
63+
- name: Install dependencies
64+
run: |
65+
pip install .[quality]
66+
67+
- name: Download Makefile from main branch
68+
run: |
69+
curl -o main_Makefile https://raw.githubusercontent.com/huggingface/diffusers/main/Makefile
70+
71+
- name: Compare Makefiles
72+
run: |
73+
if ! diff -q main_Makefile Makefile; then
74+
echo "Error: The Makefile has changed. Please ensure it matches the main branch."
75+
exit 1
76+
fi
77+
echo "No changes in Makefile. Proceeding..."
78+
rm -rf main_Makefile
79+
80+
- name: Run make style and make quality
81+
run: |
82+
make style && make quality
83+
84+
- name: Commit and push changes
85+
id: commit_and_push
86+
env:
87+
HEADREPOFULLNAME: ${{ steps.pr_info.outputs.headRepoFullName }}
88+
HEADREF: ${{ steps.pr_info.outputs.headRef }}
89+
PRNUMBER: ${{ steps.pr_info.outputs.prNumber }}
90+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
91+
run: |
92+
echo "HEADREPOFULLNAME: ${{ env.HEADREPOFULLNAME }}, HEADREF: ${{ env.HEADREF }}"
93+
# Configure git with the Actions bot user
94+
git config user.name "github-actions[bot]"
95+
git config user.email "github-actions[bot]@users.noreply.github.com"
96+
97+
# Make sure your 'origin' remote is set to the contributor's fork
98+
git remote set-url origin "https://x-access-token:${GITHUB_TOKEN}@github.com/${{ env.HEADREPOFULLNAME }}.git"
99+
100+
# If there are changes after running style/quality, commit them
101+
if [ -n "$(git status --porcelain)" ]; then
102+
git add .
103+
git commit -m "Apply style fixes"
104+
# Push to the original contributor's forked branch
105+
git push origin HEAD:${{ env.HEADREF }}
106+
echo "changes_pushed=true" >> $GITHUB_OUTPUT
107+
else
108+
echo "No changes to commit."
109+
echo "changes_pushed=false" >> $GITHUB_OUTPUT
110+
fi
111+
112+
- name: Comment on PR with workflow run link
113+
if: steps.commit_and_push.outputs.changes_pushed == 'true'
114+
uses: actions/github-script@v6
115+
with:
116+
script: |
117+
const prNumber = parseInt(process.env.prNumber, 10);
118+
const runUrl = `${process.env.GITHUB_SERVER_URL}/${process.env.GITHUB_REPOSITORY}/actions/runs/${process.env.GITHUB_RUN_ID}`
119+
120+
await github.rest.issues.createComment({
121+
owner: context.repo.owner,
122+
repo: context.repo.repo,
123+
issue_number: prNumber,
124+
body: `Style fixes have been applied. [View the workflow run here](${runUrl}).`
125+
});
126+
env:
127+
prNumber: ${{ steps.pr_info.outputs.prNumber }}

.github/workflows/pr_tests.yml

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,8 @@ name: Fast tests for PRs
22

33
on:
44
pull_request:
5-
branches:
6-
- main
5+
branches: [main]
6+
types: [synchronize]
77
paths:
88
- "src/diffusers/**.py"
99
- "benchmarks/**.py"
@@ -64,6 +64,7 @@ jobs:
6464
run: |
6565
python utils/check_copies.py
6666
python utils/check_dummies.py
67+
python utils/check_support_list.py
6768
make deps_table_check_updated
6869
- name: Check if failure
6970
if: ${{ failure() }}
@@ -120,7 +121,8 @@ jobs:
120121
run: |
121122
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
122123
python -m uv pip install -e [quality,test]
123-
python -m uv pip install accelerate
124+
pip uninstall transformers -y && python -m uv pip install -U transformers@git+https://github.com/huggingface/transformers.git --no-deps
125+
pip uninstall accelerate -y && python -m uv pip install -U accelerate@git+https://github.com/huggingface/accelerate.git --no-deps
124126
125127
- name: Environment
126128
run: |

.github/workflows/push_tests.yml

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,13 @@
11
name: Fast GPU Tests on main
22

33
on:
4+
pull_request:
5+
branches: main
6+
paths:
7+
- "src/diffusers/models/modeling_utils.py"
8+
- "src/diffusers/models/model_loading_utils.py"
9+
- "src/diffusers/pipelines/pipeline_utils.py"
10+
- "src/diffusers/pipeline_loading_utils.py"
411
workflow_dispatch:
512
push:
613
branches:
@@ -160,6 +167,7 @@ jobs:
160167
path: reports
161168

162169
flax_tpu_tests:
170+
if: ${{ github.event_name != 'pull_request' }}
163171
name: Flax TPU Tests
164172
runs-on:
165173
group: gcp-ct5lp-hightpu-8t
@@ -208,6 +216,7 @@ jobs:
208216
path: reports
209217

210218
onnx_cuda_tests:
219+
if: ${{ github.event_name != 'pull_request' }}
211220
name: ONNX CUDA Tests
212221
runs-on:
213222
group: aws-g4dn-2xlarge
@@ -256,6 +265,7 @@ jobs:
256265
path: reports
257266

258267
run_torch_compile_tests:
268+
if: ${{ github.event_name != 'pull_request' }}
259269
name: PyTorch Compile CUDA tests
260270

261271
runs-on:
@@ -299,6 +309,7 @@ jobs:
299309
path: reports
300310

301311
run_xformers_tests:
312+
if: ${{ github.event_name != 'pull_request' }}
302313
name: PyTorch xformers CUDA tests
303314

304315
runs-on:
@@ -349,7 +360,6 @@ jobs:
349360
container:
350361
image: diffusers/diffusers-pytorch-cuda
351362
options: --gpus 0 --shm-size "16gb" --ipc host
352-
353363
steps:
354364
- name: Checkout diffusers
355365
uses: actions/checkout@v3
@@ -359,7 +369,6 @@ jobs:
359369
- name: NVIDIA-SMI
360370
run: |
361371
nvidia-smi
362-
363372
- name: Install dependencies
364373
run: |
365374
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"

.github/workflows/trufflehog.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,6 @@ jobs:
1313
fetch-depth: 0
1414
- name: Secret Scanning
1515
uses: trufflesecurity/trufflehog@main
16+
with:
17+
extra_args: --results=verified,unknown
18+

docs/source/en/_toctree.yml

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,8 @@
8989
title: Kandinsky
9090
- local: using-diffusers/ip_adapter
9191
title: IP-Adapter
92+
- local: using-diffusers/omnigen
93+
title: OmniGen
9294
- local: using-diffusers/pag
9395
title: PAG
9496
- local: using-diffusers/controlnet
@@ -276,6 +278,8 @@
276278
title: ConsisIDTransformer3DModel
277279
- local: api/models/cogview3plus_transformer2d
278280
title: CogView3PlusTransformer2DModel
281+
- local: api/models/cogview4_transformer2d
282+
title: CogView4Transformer2DModel
279283
- local: api/models/dit_transformer2d
280284
title: DiTTransformer2DModel
281285
- local: api/models/flux_transformer
@@ -288,10 +292,14 @@
288292
title: LatteTransformer3DModel
289293
- local: api/models/lumina_nextdit2d
290294
title: LuminaNextDiT2DModel
295+
- local: api/models/lumina2_transformer2d
296+
title: Lumina2Transformer2DModel
291297
- local: api/models/ltx_video_transformer3d
292298
title: LTXVideoTransformer3DModel
293299
- local: api/models/mochi_transformer3d
294300
title: MochiTransformer3DModel
301+
- local: api/models/omnigen_transformer
302+
title: OmniGenTransformer2DModel
295303
- local: api/models/pixart_transformer2d
296304
title: PixArtTransformer2DModel
297305
- local: api/models/prior_transformer
@@ -376,6 +384,8 @@
376384
title: CogVideoX
377385
- local: api/pipelines/cogview3
378386
title: CogView3
387+
- local: api/pipelines/cogview4
388+
title: CogView4
379389
- local: api/pipelines/consisid
380390
title: ConsisID
381391
- local: api/pipelines/consistency_models
@@ -438,6 +448,8 @@
438448
title: LEDITS++
439449
- local: api/pipelines/ltx_video
440450
title: LTXVideo
451+
- local: api/pipelines/lumina2
452+
title: Lumina 2.0
441453
- local: api/pipelines/lumina
442454
title: Lumina-T2X
443455
- local: api/pipelines/marigold
@@ -448,6 +460,8 @@
448460
title: MultiDiffusion
449461
- local: api/pipelines/musicldm
450462
title: MusicLDM
463+
- local: api/pipelines/omnigen
464+
title: OmniGen
451465
- local: api/pipelines/pag
452466
title: PAG
453467
- local: api/pipelines/paint_by_example
@@ -598,6 +612,8 @@
598612
title: Attention Processor
599613
- local: api/activations
600614
title: Custom activation functions
615+
- local: api/cache
616+
title: Caching methods
601617
- local: api/normalization
602618
title: Custom normalization layers
603619
- local: api/utilities

docs/source/en/api/activations.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,16 @@ Customized activation functions for supporting various models in 🤗 Diffusers.
2525
## ApproximateGELU
2626

2727
[[autodoc]] models.activations.ApproximateGELU
28+
29+
30+
## SwiGLU
31+
32+
[[autodoc]] models.activations.SwiGLU
33+
34+
## FP32SiLU
35+
36+
[[autodoc]] models.activations.FP32SiLU
37+
38+
## LinearActivation
39+
40+
[[autodoc]] models.activations.LinearActivation

docs/source/en/api/attnprocessor.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,3 +147,20 @@ An attention processor is a class for applying different types of attention mech
147147
## XLAFlashAttnProcessor2_0
148148

149149
[[autodoc]] models.attention_processor.XLAFlashAttnProcessor2_0
150+
151+
## XFormersJointAttnProcessor
152+
153+
[[autodoc]] models.attention_processor.XFormersJointAttnProcessor
154+
155+
## IPAdapterXFormersAttnProcessor
156+
157+
[[autodoc]] models.attention_processor.IPAdapterXFormersAttnProcessor
158+
159+
## FluxIPAdapterJointAttnProcessor2_0
160+
161+
[[autodoc]] models.attention_processor.FluxIPAdapterJointAttnProcessor2_0
162+
163+
164+
## XLAFluxFlashAttnProcessor2_0
165+
166+
[[autodoc]] models.attention_processor.XLAFluxFlashAttnProcessor2_0

docs/source/en/api/cache.md

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
<!-- Copyright 2024 The HuggingFace Team. All rights reserved.
2+
3+
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
4+
the License. You may obtain a copy of the License at
5+
6+
http://www.apache.org/licenses/LICENSE-2.0
7+
8+
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
9+
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
10+
specific language governing permissions and limitations under the License. -->
11+
12+
# Caching methods
13+
14+
## Pyramid Attention Broadcast
15+
16+
[Pyramid Attention Broadcast](https://huggingface.co/papers/2408.12588) from Xuanlei Zhao, Xiaolong Jin, Kai Wang, Yang You.
17+
18+
Pyramid Attention Broadcast (PAB) is a method that speeds up inference in diffusion models by systematically skipping attention computations between successive inference steps and reusing cached attention states. The attention states are not very different between successive inference steps. The most prominent difference is in the spatial attention blocks, not as much in the temporal attention blocks, and finally the least in the cross attention blocks. Therefore, many cross attention computation blocks can be skipped, followed by the temporal and spatial attention blocks. By combining other techniques like sequence parallelism and classifier-free guidance parallelism, PAB achieves near real-time video generation.
19+
20+
Enable PAB with [`~PyramidAttentionBroadcastConfig`] on any pipeline. For some benchmarks, refer to [this](https://github.com/huggingface/diffusers/pull/9562) pull request.
21+
22+
```python
23+
import torch
24+
from diffusers import CogVideoXPipeline, PyramidAttentionBroadcastConfig
25+
26+
pipe = CogVideoXPipeline.from_pretrained("THUDM/CogVideoX-5b", torch_dtype=torch.bfloat16)
27+
pipe.to("cuda")
28+
29+
# Increasing the value of `spatial_attention_timestep_skip_range[0]` or decreasing the value of
30+
# `spatial_attention_timestep_skip_range[1]` will decrease the interval in which pyramid attention
31+
# broadcast is active, leader to slower inference speeds. However, large intervals can lead to
32+
# poorer quality of generated videos.
33+
config = PyramidAttentionBroadcastConfig(
34+
spatial_attention_block_skip_range=2,
35+
spatial_attention_timestep_skip_range=(100, 800),
36+
current_timestep_callback=lambda: pipe.current_timestep,
37+
)
38+
pipe.transformer.enable_cache(config)
39+
```
40+
41+
### CacheMixin
42+
43+
[[autodoc]] CacheMixin
44+
45+
### PyramidAttentionBroadcastConfig
46+
47+
[[autodoc]] PyramidAttentionBroadcastConfig
48+
49+
[[autodoc]] apply_pyramid_attention_broadcast

0 commit comments

Comments
 (0)