Skip to content

Commit d95d61a

Browse files
a-r-r-o-wa-r-r-o-w
committed
Merge branch 'main' into pyramid-attention-broadcast
2 parents 3de2c18 + bf64b32 commit d95d61a

File tree

259 files changed

+15697
-2388
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

259 files changed

+15697
-2388
lines changed

‎.github/workflows/nightly_tests.yml‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -347,6 +347,64 @@ jobs:
347347
pip install slack_sdk tabulate
348348
python utils/log_reports.py >> $GITHUB_STEP_SUMMARY
349349
350+
run_nightly_quantization_tests:
351+
name: Torch quantization nightly tests
352+
strategy:
353+
fail-fast: false
354+
max-parallel: 2
355+
matrix:
356+
config:
357+
- backend: "bitsandbytes"
358+
test_location: "bnb"
359+
runs-on:
360+
group: aws-g6e-xlarge-plus
361+
container:
362+
image: diffusers/diffusers-pytorch-cuda
363+
options: --shm-size "20gb" --ipc host --gpus 0
364+
steps:
365+
- name: Checkout diffusers
366+
uses: actions/checkout@v3
367+
with:
368+
fetch-depth: 2
369+
- name: NVIDIA-SMI
370+
run: nvidia-smi
371+
- name: Install dependencies
372+
run: |
373+
python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH"
374+
python -m uv pip install -e [quality,test]
375+
python -m uv pip install -U ${{ matrix.config.backend }}
376+
python -m uv pip install pytest-reportlog
377+
- name: Environment
378+
run: |
379+
python utils/print_env.py
380+
- name: ${{ matrix.config.backend }} quantization tests on GPU
381+
env:
382+
HF_TOKEN: ${{ secrets.DIFFUSERS_HF_HUB_READ_TOKEN }}
383+
# https://pytorch.org/docs/stable/notes/randomness.html#avoiding-nondeterministic-algorithms
384+
CUBLAS_WORKSPACE_CONFIG: :16:8
385+
BIG_GPU_MEMORY: 40
386+
run: |
387+
python -m pytest -n 1 --max-worker-restart=0 --dist=loadfile \
388+
--make-reports=tests_${{ matrix.config.backend }}_torch_cuda \
389+
--report-log=tests_${{ matrix.config.backend }}_torch_cuda.log \
390+
tests/quantization/${{ matrix.config.test_location }}
391+
- name: Failure short reports
392+
if: ${{ failure() }}
393+
run: |
394+
cat reports/tests_${{ matrix.config.backend }}_torch_cuda_stats.txt
395+
cat reports/tests_${{ matrix.config.backend }}_torch_cuda_failures_short.txt
396+
- name: Test suite reports artifacts
397+
if: ${{ always() }}
398+
uses: actions/upload-artifact@v4
399+
with:
400+
name: torch_cuda_${{ matrix.config.backend }}_reports
401+
path: reports
402+
- name: Generate Report and Notify Channel
403+
if: always()
404+
run: |
405+
pip install slack_sdk tabulate
406+
python utils/log_reports.py >> $GITHUB_STEP_SUMMARY
407+
350408
# M1 runner currently not well supported
351409
# TODO: (Dhruv) add these back when we setup better testing for Apple Silicon
352410
# run_nightly_tests_apple_m1:

‎README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -112,9 +112,9 @@ Check out the [Quickstart](https://huggingface.co/docs/diffusers/quicktour) to l
112112
| **Documentation** | **What can I learn?** |
113113
|---------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
114114
| [Tutorial](https://huggingface.co/docs/diffusers/tutorials/tutorial_overview) | A basic crash course for learning how to use the library's most important features like using models and schedulers to build your own diffusion system, and training your own diffusion model. |
115-
| [Loading](https://huggingface.co/docs/diffusers/using-diffusers/loading_overview) | Guides for how to load and configure all the components (pipelines, models, and schedulers) of the library, as well as how to use different schedulers. |
116-
| [Pipelines for inference](https://huggingface.co/docs/diffusers/using-diffusers/pipeline_overview) | Guides for how to use pipelines for different inference tasks, batched generation, controlling generated outputs and randomness, and how to contribute a pipeline to the library. |
117-
| [Optimization](https://huggingface.co/docs/diffusers/optimization/opt_overview) | Guides for how to optimize your diffusion model to run faster and consume less memory. |
115+
| [Loading](https://huggingface.co/docs/diffusers/using-diffusers/loading) | Guides for how to load and configure all the components (pipelines, models, and schedulers) of the library, as well as how to use different schedulers. |
116+
| [Pipelines for inference](https://huggingface.co/docs/diffusers/using-diffusers/overview_techniques) | Guides for how to use pipelines for different inference tasks, batched generation, controlling generated outputs and randomness, and how to contribute a pipeline to the library. |
117+
| [Optimization](https://huggingface.co/docs/diffusers/optimization/fp16) | Guides for how to optimize your diffusion model to run faster and consume less memory. |
118118
| [Training](https://huggingface.co/docs/diffusers/training/overview) | Guides for how to train a diffusion model for different tasks with different training techniques. |
119119
## Contribution
120120

‎docker/diffusers-onnxruntime-cuda/Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ ENV PATH="/opt/venv/bin:$PATH"
2828
# pre-install the heavy dependencies (these can later be overridden by the deps from setup.py)
2929
RUN python3.10 -m pip install --no-cache-dir --upgrade pip uv==0.1.11 && \
3030
python3.10 -m uv pip install --no-cache-dir \
31-
"torch<2.5.0" \
31+
torch \
3232
torchvision \
3333
torchaudio \
3434
"onnxruntime-gpu>=1.13.1" \

‎docker/diffusers-pytorch-compile-cuda/Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ ENV PATH="/opt/venv/bin:$PATH"
2929
# pre-install the heavy dependencies (these can later be overridden by the deps from setup.py)
3030
RUN python3.10 -m pip install --no-cache-dir --upgrade pip uv==0.1.11 && \
3131
python3.10 -m uv pip install --no-cache-dir \
32-
"torch<2.5.0" \
32+
torch \
3333
torchvision \
3434
torchaudio \
3535
invisible_watermark && \

‎docker/diffusers-pytorch-cpu/Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ ENV PATH="/opt/venv/bin:$PATH"
2929
# pre-install the heavy dependencies (these can later be overridden by the deps from setup.py)
3030
RUN python3.10 -m pip install --no-cache-dir --upgrade pip uv==0.1.11 && \
3131
python3.10 -m uv pip install --no-cache-dir \
32-
"torch<2.5.0" \
32+
torch \
3333
torchvision \
3434
torchaudio \
3535
invisible_watermark \

‎docker/diffusers-pytorch-cuda/Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ ENV PATH="/opt/venv/bin:$PATH"
2929
# pre-install the heavy dependencies (these can later be overridden by the deps from setup.py)
3030
RUN python3.10 -m pip install --no-cache-dir --upgrade pip uv==0.1.11 && \
3131
python3.10 -m uv pip install --no-cache-dir \
32-
"torch<2.5.0" \
32+
torch \
3333
torchvision \
3434
torchaudio \
3535
invisible_watermark && \

‎docker/diffusers-pytorch-xformers-cuda/Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ ENV PATH="/opt/venv/bin:$PATH"
2929
# pre-install the heavy dependencies (these can later be overridden by the deps from setup.py)
3030
RUN python3.10 -m pip install --no-cache-dir --upgrade pip uv==0.1.11 && \
3131
python3.10 -m pip install --no-cache-dir \
32-
"torch<2.5.0" \
32+
torch \
3333
torchvision \
3434
torchaudio \
3535
invisible_watermark && \

‎docs/source/en/_toctree.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,8 @@
5555
- sections:
5656
- local: using-diffusers/overview_techniques
5757
title: Overview
58+
- local: using-diffusers/create_a_server
59+
title: Create a server
5860
- local: training/distributed_inference
5961
title: Distributed inference
6062
- local: using-diffusers/merge_loras

‎docs/source/en/api/pipelines/cogvideox.md‎

Lines changed: 24 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -29,16 +29,32 @@ Make sure to check out the Schedulers [guide](../../using-diffusers/schedulers.m
2929

3030
This pipeline was contributed by [zRzRzRzRzRzRzR](https://github.com/zRzRzRzRzRzRzR). The original codebase can be found [here](https://huggingface.co/THUDM). The original weights can be found under [hf.co/THUDM](https://huggingface.co/THUDM).
3131

32-
There are two models available that can be used with the text-to-video and video-to-video CogVideoX pipelines:
33-
- [`THUDM/CogVideoX-2b`](https://huggingface.co/THUDM/CogVideoX-2b): The recommended dtype for running this model is `fp16`.
34-
- [`THUDM/CogVideoX-5b`](https://huggingface.co/THUDM/CogVideoX-5b): The recommended dtype for running this model is `bf16`.
32+
There are three official CogVideoX checkpoints for text-to-video and video-to-video.
3533

36-
There is one model available that can be used with the image-to-video CogVideoX pipeline:
37-
- [`THUDM/CogVideoX-5b-I2V`](https://huggingface.co/THUDM/CogVideoX-5b-I2V): The recommended dtype for running this model is `bf16`.
34+
| checkpoints | recommended inference dtype |
35+
|:---:|:---:|
36+
| [`THUDM/CogVideoX-2b`](https://huggingface.co/THUDM/CogVideoX-2b) | torch.float16 |
37+
| [`THUDM/CogVideoX-5b`](https://huggingface.co/THUDM/CogVideoX-5b) | torch.bfloat16 |
38+
| [`THUDM/CogVideoX1.5-5b`](https://huggingface.co/THUDM/CogVideoX1.5-5b) | torch.bfloat16 |
3839

39-
There are two models that support pose controllable generation (by the [Alibaba-PAI](https://huggingface.co/alibaba-pai) team):
40-
- [`alibaba-pai/CogVideoX-Fun-V1.1-2b-Pose`](https://huggingface.co/alibaba-pai/CogVideoX-Fun-V1.1-2b-Pose): The recommended dtype for running this model is `bf16`.
41-
- [`alibaba-pai/CogVideoX-Fun-V1.1-5b-Pose`](https://huggingface.co/alibaba-pai/CogVideoX-Fun-V1.1-5b-Pose): The recommended dtype for running this model is `bf16`.
40+
There are two official CogVideoX checkpoints available for image-to-video.
41+
42+
| checkpoints | recommended inference dtype |
43+
|:---:|:---:|
44+
| [`THUDM/CogVideoX-5b-I2V`](https://huggingface.co/THUDM/CogVideoX-5b-I2V) | torch.bfloat16 |
45+
| [`THUDM/CogVideoX-1.5-5b-I2V`](https://huggingface.co/THUDM/CogVideoX-1.5-5b-I2V) | torch.bfloat16 |
46+
47+
For the CogVideoX 1.5 series:
48+
- Text-to-video (T2V) works best at a resolution of 1360x768 because it was trained with that specific resolution.
49+
- Image-to-video (I2V) works for multiple resolutions. The width can vary from 768 to 1360, but the height must be 768. The height/width must be divisible by 16.
50+
- Both T2V and I2V models support generation with 81 and 161 frames and work best at this value. Exporting videos at 16 FPS is recommended.
51+
52+
There are two official CogVideoX checkpoints that support pose controllable generation (by the [Alibaba-PAI](https://huggingface.co/alibaba-pai) team).
53+
54+
| checkpoints | recommended inference dtype |
55+
|:---:|:---:|
56+
| [`alibaba-pai/CogVideoX-Fun-V1.1-2b-Pose`](https://huggingface.co/alibaba-pai/CogVideoX-Fun-V1.1-2b-Pose) | torch.bfloat16 |
57+
| [`alibaba-pai/CogVideoX-Fun-V1.1-5b-Pose`](https://huggingface.co/alibaba-pai/CogVideoX-Fun-V1.1-5b-Pose) | torch.bfloat16 |
4258

4359
## Inference
4460

‎docs/source/en/api/pipelines/controlnet_sd3.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ This controlnet code is mainly implemented by [The InstantX Team](https://huggin
2828
| ControlNet type | Developer | Link |
2929
| -------- | ---------- | ---- |
3030
| Canny | [The InstantX Team](https://huggingface.co/InstantX) | [Link](https://huggingface.co/InstantX/SD3-Controlnet-Canny) |
31+
| Depth | [The InstantX Team](https://huggingface.co/InstantX) | [Link](https://huggingface.co/InstantX/SD3-Controlnet-Depth) |
3132
| Pose | [The InstantX Team](https://huggingface.co/InstantX) | [Link](https://huggingface.co/InstantX/SD3-Controlnet-Pose) |
3233
| Tile | [The InstantX Team](https://huggingface.co/InstantX) | [Link](https://huggingface.co/InstantX/SD3-Controlnet-Tile) |
3334
| Inpainting | [The AlimamaCreative Team](https://huggingface.co/alimama-creative) | [link](https://huggingface.co/alimama-creative/SD3-Controlnet-Inpainting) |

0 commit comments

Comments
 (0)