Skip to content

Commit 2d8a41c

Browse files
yiyixuxuyitonghwan-x-aia-r-r-o-w
authored
[Alibaba Wan Team] continue on huggingface#10921 Wan2.1 (huggingface#10922)
* Add wanx pipeline, model and example * wanx_merged_v1 * change WanX into Wan * fix i2v fp32 oom error Link: https://code.alibaba-inc.com/open_wanx2/diffusers/codereview/20607813 * support t2v load fp32 ckpt * add example * final merge v1 * Update autoencoder_kl_wan.py * up * update middle, test up_block * up up * one less nn.sequential * up more * up * more * [refactor] [wip] Wan transformer/pipeline (huggingface#10926) * update * update * refactor rope * refactor pipeline * make fix-copies * add transformer test * update * update * make style * update tests * tests * conversion script * conversion script * update * docs * remove unused code * fix _toctree.yml * update dtype * fix test * fix tests: scale * up * more * Apply suggestions from code review * Apply suggestions from code review * style * Update scripts/convert_wan_to_diffusers.py * update docs * fix --------- Co-authored-by: Yitong Huang <[email protected]> Co-authored-by: 亚森 <[email protected]> Co-authored-by: Aryan <[email protected]>
1 parent 7007feb commit 2d8a41c

26 files changed

+3700
-3
lines changed

‎docs/source/en/_toctree.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -314,6 +314,8 @@
314314
title: Transformer2DModel
315315
- local: api/models/transformer_temporal
316316
title: TransformerTemporalModel
317+
- local: api/models/wan_transformer_3d
318+
title: WanTransformer3DModel
317319
title: Transformers
318320
- sections:
319321
- local: api/models/stable_cascade_unet
@@ -344,6 +346,8 @@
344346
title: AutoencoderKLLTXVideo
345347
- local: api/models/autoencoderkl_mochi
346348
title: AutoencoderKLMochi
349+
- local: api/models/autoencoder_kl_wan
350+
title: AutoencoderKLWan
347351
- local: api/models/asymmetricautoencoderkl
348352
title: AsymmetricAutoencoderKL
349353
- local: api/models/autoencoder_dc
@@ -534,6 +538,8 @@
534538
title: UniDiffuser
535539
- local: api/pipelines/value_guided_sampling
536540
title: Value-guided sampling
541+
- local: api/pipelines/wan
542+
title: Wan
537543
- local: api/pipelines/wuerstchen
538544
title: Wuerstchen
539545
title: Pipelines

‎docs/source/en/api/models/autoencoder_kl_wan.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
<!-- Copyright 2024 The HuggingFace Team. All rights reserved.
2+
3+
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
4+
the License. You may obtain a copy of the License at
5+
6+
http://www.apache.org/licenses/LICENSE-2.0
7+
8+
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
9+
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
10+
specific language governing permissions and limitations under the License. -->
11+
12+
# AutoencoderKLWan
13+
14+
The 3D variational autoencoder (VAE) model with KL loss used in [Wan 2.1](https://github.com/Wan-Video/Wan2.1) by the Alibaba Wan Team.
15+
16+
The model can be loaded with the following code snippet.
17+
18+
```python
19+
from diffusers import AutoencoderKLWan
20+
21+
vae = AutoencoderKLWan.from_pretrained("Wan-AI/Wan2.1-T2V-1.3B-Diffusers", subfolder="vae", torch_dtype=torch.float32)
22+
```
23+
24+
## AutoencoderKLWan
25+
26+
[[autodoc]] AutoencoderKLWan
27+
- decode
28+
- all
29+
30+
## DecoderOutput
31+
32+
[[autodoc]] models.autoencoders.vae.DecoderOutput

‎docs/source/en/api/models/wan_transformer_3d.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
<!-- Copyright 2024 The HuggingFace Team. All rights reserved.
2+
3+
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
4+
the License. You may obtain a copy of the License at
5+
6+
http://www.apache.org/licenses/LICENSE-2.0
7+
8+
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
9+
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
10+
specific language governing permissions and limitations under the License. -->
11+
12+
# WanTransformer3DModel
13+
14+
A Diffusion Transformer model for 3D video-like data was introduced in [Wan 2.1](https://github.com/Wan-Video/Wan2.1) by the Alibaba Wan Team.
15+
16+
The model can be loaded with the following code snippet.
17+
18+
```python
19+
from diffusers import WanTransformer3DModel
20+
21+
transformer = WanTransformer3DModel.from_pretrained("Wan-AI/Wan2.1-T2V-1.3B-Diffusers", subfolder="transformer", torch_dtype=torch.bfloat16)
22+
```
23+
24+
## WanTransformer3DModel
25+
26+
[[autodoc]] WanTransformer3DModel
27+
28+
## Transformer2DModelOutput
29+
30+
[[autodoc]] models.modeling_outputs.Transformer2DModelOutput

‎docs/source/en/api/pipelines/wan.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
<!-- Copyright 2024 The HuggingFace Team. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License. -->
14+
15+
# Wan
16+
17+
[Wan 2.1](https://github.com/Wan-Video/Wan2.1) by the Alibaba Wan Team.
18+
19+
<!-- TODO(aryan): update abstract once paper is out -->
20+
21+
<Tip>
22+
23+
Make sure to check out the Schedulers [guide](../../using-diffusers/schedulers) to learn how to explore the tradeoff between scheduler speed and quality, and see the [reuse components across pipelines](../../using-diffusers/loading#reuse-a-pipeline) section to learn how to efficiently load the same components into multiple pipelines.
24+
25+
</Tip>
26+
27+
Recommendations for inference:
28+
- VAE in `torch.float32` for better decoding quality.
29+
- `num_frames` should be of the form `4 * k + 1`, for example `49` or `81`.
30+
- For smaller resolution videos, try lower values of `shift` (between `2.0` to `5.0`) in the [Scheduler](https://huggingface.co/docs/diffusers/main/en/api/schedulers/flow_match_euler_discrete#diffusers.FlowMatchEulerDiscreteScheduler.shift). For larger resolution videos, try higher values (between `7.0` and `12.0`). The default value is `3.0` for Wan.
31+
32+
### Using a custom scheduler
33+
34+
Wan can be used with many different schedulers, each with their own benefits regarding speed and generation quality. By default, Wan uses the `UniPCMultistepScheduler(prediction_type="flow_prediction", use_flow_sigmas=True, flow_shift=3.0)` scheduler. You can use a different scheduler as follows:
35+
36+
```python
37+
from diffusers import FlowMatchEulerDiscreteScheduler, UniPCMultistepScheduler, WanPipeline
38+
39+
scheduler_a = FlowMatchEulerDiscreteScheduler(shift=5.0)
40+
scheduler_b = UniPCMultistepScheduler(prediction_type="flow_prediction", use_flow_sigmas=True, flow_shift=4.0)
41+
42+
pipe = WanPipeline.from_pretrained("Wan-AI/Wan2.1-T2V-1.3B-Diffusers", scheduler=<CUSTOM_SCHEDULER_HERE>)
43+
44+
# or,
45+
pipe.scheduler = <CUSTOM_SCHEDULER_HERE>
46+
```
47+
48+
## WanPipeline
49+
50+
[[autodoc]] WanPipeline
51+
- all
52+
- __call__
53+
54+
## WanImageToVideoPipeline
55+
56+
[[autodoc]] WanImageToVideoPipeline
57+
- all
58+
- __call__
59+
60+
## WanPipelineOutput
61+
62+
[[autodoc]] pipelines.wan.pipeline_output.WanPipelineOutput

0 commit comments

Comments
 (0)