Skip to content

Commit 94eb110

Browse files
committed
implement as a pipeline, and add documentation
1 parent a8a5c5e commit 94eb110

File tree

2 files changed

+411
-187
lines changed

2 files changed

+411
-187
lines changed

examples/community/README.md

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,7 @@ PIXART-α Controlnet pipeline | Implementation of the controlnet model for pixar
8282
| [🪆Matryoshka Diffusion Models](https://huggingface.co/papers/2310.15111) | A diffusion process that denoises inputs at multiple resolutions jointly and uses a NestedUNet architecture where features and parameters for small scale inputs are nested within those of the large scales. See [original codebase](https://github.com/apple/ml-mdm). | [🪆Matryoshka Diffusion Models](#matryoshka-diffusion-models) | [![Hugging Face Space](https://img.shields.io/badge/🤗%20Hugging%20Face-Space-yellow)](https://huggingface.co/spaces/pcuenq/mdm) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/tolgacangoz/1f54875fc7aeaabcf284ebde64820966/matryoshka_hf.ipynb) | [M. Tolga Cangöz](https://github.com/tolgacangoz) |
8383
| Stable Diffusion XL Attentive Eraser Pipeline |[[AAAI2025 Oral] Attentive Eraser](https://github.com/Anonym0u3/AttentiveEraser) is a novel tuning-free method that enhances object removal capabilities in pre-trained diffusion models.|[Stable Diffusion XL Attentive Eraser Pipeline](#stable-diffusion-xl-attentive-eraser-pipeline)|-|[Wenhao Sun](https://github.com/Anonym0u3) and [Benlei Cui](https://github.com/Benny079)|
8484
| Perturbed-Attention Guidance |StableDiffusionPAGPipeline is a modification of StableDiffusionPipeline to support Perturbed-Attention Guidance (PAG).|[Perturbed-Attention Guidance](#perturbed-attention-guidance)|[Notebook](https://github.com/huggingface/notebooks/blob/main/diffusers/perturbed_attention_guidance.ipynb)|[Hyoungwon Cho](https://github.com/HyoungwonCho)|
85+
| CogVideoX DDIM Inversion Pipeline | Implementation of DDIM inversion and guided attention-based editing denoising process on CogVideoX. | [CogVideoX DDIM Inversion Pipeline](#cogvideox-ddim-inversion-pipeline) | - | [LittleNyima](https://github.com/LittleNyima) |
8586

8687
To load a custom pipeline you just need to pass the `custom_pipeline` argument to `DiffusionPipeline`, as one of the files in `diffusers/examples/community`. Feel free to send a PR with your own pipelines, we will merge them quickly.
8788

@@ -5124,3 +5125,39 @@ with torch.no_grad():
51245125

51255126
In the folder examples/pixart there is also a script that can be used to train new models.
51265127
Please check the script `train_controlnet_hf_diffusers.sh` on how to start the training.
5128+
5129+
# CogVideoX DDIM Inversion Pipeline
5130+
5131+
This implementation performs DDIM inversion on the video based on CogVideoX and uses guided attention to reconstruct or edit the inversion latents.
5132+
5133+
## Example Usage
5134+
5135+
```python
5136+
import torch
5137+
5138+
from examples.community.cogvideox_ddim_inversion import CogVideoXPipelineForDDIMInversion
5139+
5140+
5141+
# Load pretrained pipeline
5142+
pipeline = CogVideoXPipelineForDDIMInversion.from_pretrained(
5143+
"THUDM/CogVideoX1.5-5B",
5144+
torch_dtype=torch.bfloat16,
5145+
).to("cuda")
5146+
5147+
# Run DDIM inversion, and the videos will be generated in the output_path
5148+
output = pipeline_for_inversion(
5149+
prompt="prompt that describes the edited video",
5150+
video_path="path/to/input.mp4",
5151+
guidance_scale=6.0,
5152+
num_inference_steps=50,
5153+
skip_frames_start=0,
5154+
skip_frames_end=0,
5155+
frame_sample_step=None,
5156+
max_num_frames=81,
5157+
width=720,
5158+
height=480,
5159+
seed=42,
5160+
)
5161+
pipeline.export_latents_to_video(output.inverse_latents[-1], "path/to/inverse_video.mp4", fps=8)
5162+
pipeline.export_latents_to_video(output.recon_latents[-1], "path/to/recon_video.mp4", fps=8)
5163+
```

0 commit comments

Comments
 (0)