|
| 1 | +<!-- Copyright 2025 The HuggingFace Team. All rights reserved. |
| 2 | +# |
| 3 | +# Licensed under the Apache License, Version 2.0 (the "License"); |
| 4 | +# you may not use this file except in compliance with the License. |
| 5 | +# You may obtain a copy of the License at |
| 6 | +# |
| 7 | +# http://www.apache.org/licenses/LICENSE-2.0 |
| 8 | +# |
| 9 | +# Unless required by applicable law or agreed to in writing, software |
| 10 | +# distributed under the License is distributed on an "AS IS" BASIS, |
| 11 | +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| 12 | +# See the License for the specific language governing permissions and |
| 13 | +# limitations under the License. --> |
| 14 | + |
| 15 | +# Framepack |
| 16 | + |
| 17 | +<div class="flex flex-wrap space-x-1"> |
| 18 | + <img alt="LoRA" src="https://img.shields.io/badge/LoRA-d8b4fe?style=flat"/> |
| 19 | +</div> |
| 20 | + |
| 21 | +[Packing Input Frame Context in Next-Frame Prediction Models for Video Generation](https://arxiv.org/abs/2504.12626) by Lvmin Zhang and Maneesh Agrawala. |
| 22 | + |
| 23 | +*We present a neural network structure, FramePack, to train next-frame (or next-frame-section) prediction models for video generation. The FramePack compresses input frames to make the transformer context length a fixed number regardless of the video length. As a result, we are able to process a large number of frames using video diffusion with computation bottleneck similar to image diffusion. This also makes the training video batch sizes significantly higher (batch sizes become comparable to image diffusion training). We also propose an anti-drifting sampling method that generates frames in inverted temporal order with early-established endpoints to avoid exposure bias (error accumulation over iterations). Finally, we show that existing video diffusion models can be finetuned with FramePack, and their visual quality may be improved because the next-frame prediction supports more balanced diffusion schedulers with less extreme flow shift timesteps.* |
| 24 | + |
| 25 | +<Tip> |
| 26 | + |
| 27 | +Make sure to check out the Schedulers [guide](../../using-diffusers/schedulers) to learn how to explore the tradeoff between scheduler speed and quality, and see the [reuse components across pipelines](../../using-diffusers/loading#reuse-a-pipeline) section to learn how to efficiently load the same components into multiple pipelines. |
| 28 | + |
| 29 | +</Tip> |
| 30 | + |
| 31 | +## Available models |
| 32 | + |
| 33 | +| Model name | Description | |
| 34 | +|:---|:---| |
| 35 | +- [`lllyasviel/FramePackI2V_HY`](https://huggingface.co/lllyasviel/FramePackI2V_HY) | Trained with the "inverted anti-drifting" strategy as described in the paper. Inference requires setting `sampling_type="inverted_anti_drifting"` when running the pipeline. | |
| 36 | +- [`lllyasviel/FramePack_F1_I2V_HY_20250503`](https://huggingface.co/lllyasviel/FramePack_F1_I2V_HY_20250503) | Trained with a novel anti-drifting strategy but inference is performed in "vanilla" strategy as described in the paper. Inference requires setting `sampling_type="vanilla"` when running the pipeline. | |
| 37 | + |
| 38 | +## Usage |
| 39 | + |
| 40 | +Refer to the pipeline documentation for basic usage examples. The following section contains examples of offloading, different sampling methods, quantization, and more. |
| 41 | + |
| 42 | +### First and last frame to video |
| 43 | + |
| 44 | +The following example shows how to use Framepack with start and end image controls, using the inverted anti-drifiting sampling model. |
| 45 | + |
| 46 | +```python |
| 47 | +import torch |
| 48 | +from diffusers import HunyuanVideoFramepackPipeline, HunyuanVideoFramepackTransformer3DModel |
| 49 | +from diffusers.utils import export_to_video, load_image |
| 50 | +from transformers import SiglipImageProcessor, SiglipVisionModel |
| 51 | + |
| 52 | +transformer = HunyuanVideoFramepackTransformer3DModel.from_pretrained( |
| 53 | + "lllyasviel/FramePackI2V_HY", torch_dtype=torch.bfloat16 |
| 54 | +) |
| 55 | +feature_extractor = SiglipImageProcessor.from_pretrained( |
| 56 | + "lllyasviel/flux_redux_bfl", subfolder="feature_extractor" |
| 57 | +) |
| 58 | +image_encoder = SiglipVisionModel.from_pretrained( |
| 59 | + "lllyasviel/flux_redux_bfl", subfolder="image_encoder", torch_dtype=torch.float16 |
| 60 | +) |
| 61 | +pipe = HunyuanVideoFramepackPipeline.from_pretrained( |
| 62 | + "hunyuanvideo-community/HunyuanVideo", |
| 63 | + transformer=transformer, |
| 64 | + feature_extractor=feature_extractor, |
| 65 | + image_encoder=image_encoder, |
| 66 | + torch_dtype=torch.float16, |
| 67 | +) |
| 68 | + |
| 69 | +# Enable memory optimizations |
| 70 | +pipe.enable_model_cpu_offload() |
| 71 | +pipe.vae.enable_tiling() |
| 72 | + |
| 73 | +prompt = "CG animation style, a small blue bird takes off from the ground, flapping its wings. The bird's feathers are delicate, with a unique pattern on its chest. The background shows a blue sky with white clouds under bright sunshine. The camera follows the bird upward, capturing its flight and the vastness of the sky from a close-up, low-angle perspective." |
| 74 | +first_image = load_image( |
| 75 | + "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flf2v_input_first_frame.png" |
| 76 | +) |
| 77 | +last_image = load_image( |
| 78 | + "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flf2v_input_last_frame.png" |
| 79 | +) |
| 80 | +output = pipe( |
| 81 | + image=first_image, |
| 82 | + last_image=last_image, |
| 83 | + prompt=prompt, |
| 84 | + height=512, |
| 85 | + width=512, |
| 86 | + num_frames=91, |
| 87 | + num_inference_steps=30, |
| 88 | + guidance_scale=9.0, |
| 89 | + generator=torch.Generator().manual_seed(0), |
| 90 | + sampling_type="inverted_anti_drifting", |
| 91 | +).frames[0] |
| 92 | +export_to_video(output, "output.mp4", fps=30) |
| 93 | +``` |
| 94 | + |
| 95 | +### Vanilla sampling |
| 96 | + |
| 97 | +The following example shows how to use Framepack with the F1 model trained with vanilla sampling but new regulation approach for anti-drifting. |
| 98 | + |
| 99 | +```python |
| 100 | +import torch |
| 101 | +from diffusers import HunyuanVideoFramepackPipeline, HunyuanVideoFramepackTransformer3DModel |
| 102 | +from diffusers.utils import export_to_video, load_image |
| 103 | +from transformers import SiglipImageProcessor, SiglipVisionModel |
| 104 | + |
| 105 | +transformer = HunyuanVideoFramepackTransformer3DModel.from_pretrained( |
| 106 | + "lllyasviel/FramePack_F1_I2V_HY_20250503", torch_dtype=torch.bfloat16 |
| 107 | +) |
| 108 | +feature_extractor = SiglipImageProcessor.from_pretrained( |
| 109 | + "lllyasviel/flux_redux_bfl", subfolder="feature_extractor" |
| 110 | +) |
| 111 | +image_encoder = SiglipVisionModel.from_pretrained( |
| 112 | + "lllyasviel/flux_redux_bfl", subfolder="image_encoder", torch_dtype=torch.float16 |
| 113 | +) |
| 114 | +pipe = HunyuanVideoFramepackPipeline.from_pretrained( |
| 115 | + "hunyuanvideo-community/HunyuanVideo", |
| 116 | + transformer=transformer, |
| 117 | + feature_extractor=feature_extractor, |
| 118 | + image_encoder=image_encoder, |
| 119 | + torch_dtype=torch.float16, |
| 120 | +) |
| 121 | + |
| 122 | +# Enable memory optimizations |
| 123 | +pipe.enable_model_cpu_offload() |
| 124 | +pipe.vae.enable_tiling() |
| 125 | + |
| 126 | +image = load_image( |
| 127 | + "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/penguin.png" |
| 128 | +) |
| 129 | +output = pipe( |
| 130 | + image=image, |
| 131 | + prompt="A penguin dancing in the snow", |
| 132 | + height=832, |
| 133 | + width=480, |
| 134 | + num_frames=91, |
| 135 | + num_inference_steps=30, |
| 136 | + guidance_scale=9.0, |
| 137 | + generator=torch.Generator().manual_seed(0), |
| 138 | + sampling_type="vanilla", |
| 139 | +).frames[0] |
| 140 | +export_to_video(output, "output.mp4", fps=30) |
| 141 | +``` |
| 142 | + |
| 143 | +### Group offloading |
| 144 | + |
| 145 | +Group offloading ([`~hooks.apply_group_offloading`]) provides aggressive memory optimizations for offloading internal parts of any model to the CPU, with possibly no additional overhead to generation time. If you have very low VRAM available, this approach may be suitable for you depending on the amount of CPU RAM available. |
| 146 | + |
| 147 | +```python |
| 148 | +import torch |
| 149 | +from diffusers import HunyuanVideoFramepackPipeline, HunyuanVideoFramepackTransformer3DModel |
| 150 | +from diffusers.hooks import apply_group_offloading |
| 151 | +from diffusers.utils import export_to_video, load_image |
| 152 | +from transformers import SiglipImageProcessor, SiglipVisionModel |
| 153 | + |
| 154 | +transformer = HunyuanVideoFramepackTransformer3DModel.from_pretrained( |
| 155 | + "lllyasviel/FramePack_F1_I2V_HY_20250503", torch_dtype=torch.bfloat16 |
| 156 | +) |
| 157 | +feature_extractor = SiglipImageProcessor.from_pretrained( |
| 158 | + "lllyasviel/flux_redux_bfl", subfolder="feature_extractor" |
| 159 | +) |
| 160 | +image_encoder = SiglipVisionModel.from_pretrained( |
| 161 | + "lllyasviel/flux_redux_bfl", subfolder="image_encoder", torch_dtype=torch.float16 |
| 162 | +) |
| 163 | +pipe = HunyuanVideoFramepackPipeline.from_pretrained( |
| 164 | + "hunyuanvideo-community/HunyuanVideo", |
| 165 | + transformer=transformer, |
| 166 | + feature_extractor=feature_extractor, |
| 167 | + image_encoder=image_encoder, |
| 168 | + torch_dtype=torch.float16, |
| 169 | +) |
| 170 | + |
| 171 | +# Enable group offloading |
| 172 | +onload_device = torch.device("cuda") |
| 173 | +offload_device = torch.device("cpu") |
| 174 | +list(map( |
| 175 | + lambda x: apply_group_offloading(x, onload_device, offload_device, offload_type="leaf_level", use_stream=True, low_cpu_mem_usage=True), |
| 176 | + [pipe.text_encoder, pipe.text_encoder_2, pipe.transformer] |
| 177 | +)) |
| 178 | +pipe.image_encoder.to(onload_device) |
| 179 | +pipe.vae.to(onload_device) |
| 180 | +pipe.vae.enable_tiling() |
| 181 | + |
| 182 | +image = load_image( |
| 183 | + "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/penguin.png" |
| 184 | +) |
| 185 | +output = pipe( |
| 186 | + image=image, |
| 187 | + prompt="A penguin dancing in the snow", |
| 188 | + height=832, |
| 189 | + width=480, |
| 190 | + num_frames=91, |
| 191 | + num_inference_steps=30, |
| 192 | + guidance_scale=9.0, |
| 193 | + generator=torch.Generator().manual_seed(0), |
| 194 | + sampling_type="vanilla", |
| 195 | +).frames[0] |
| 196 | +print(f"Max memory: {torch.cuda.max_memory_allocated() / 1024**3:.3f} GB") |
| 197 | +export_to_video(output, "output.mp4", fps=30) |
| 198 | +``` |
| 199 | + |
| 200 | +## HunyuanVideoFramepackPipeline |
| 201 | + |
| 202 | +[[autodoc]] HunyuanVideoFramepackPipeline |
| 203 | + - all |
| 204 | + - __call__ |
| 205 | + |
| 206 | +## HunyuanVideoPipelineOutput |
| 207 | + |
| 208 | +[[autodoc]] pipelines.hunyuan_video.pipeline_output.HunyuanVideoPipelineOutput |
| 209 | + |
0 commit comments