|
| 1 | +--- |
| 2 | +title: "ComfyUI Wan2.2 Fun Inp Start-End Frame Video Generation Example" |
| 3 | +description: "This article introduces how to use ComfyUI to complete the Wan2.2 Fun Inp start-end frame video generation example" |
| 4 | +sidebarTitle: "Wan2.2 Fun Inp" |
| 5 | +--- |
| 6 | + |
| 7 | +import UpdateReminder from '/snippets/tutorials/update-reminder.mdx' |
| 8 | + |
| 9 | +**Wan2.2-Fun-Inp** is a start-end frame controlled video generation model launched by Alibaba PAI team. It supports inputting **start and end frame images** to generate intermediate transition videos, providing creators with greater creative control. The model is released under the **Apache 2.0 license** and supports commercial use. |
| 10 | + |
| 11 | +**Key Features**: |
| 12 | +- **Start-End Frame Control**: Supports inputting start and end frame images to generate intermediate transition videos, enhancing video coherence and creative freedom |
| 13 | +- **High-Quality Video Generation**: Based on the Wan2.2 architecture, outputs film-level quality videos |
| 14 | +- **Multi-Resolution Support**: Supports generating videos at 512×512, 768×768, 1024×1024 and other resolutions to suit different scenarios |
| 15 | + |
| 16 | +**Model Version**: |
| 17 | +- **14B High-Performance Version**: Model size exceeds 32GB, with better results but requires high VRAM |
| 18 | + |
| 19 | +Below are the relevant model weights and code repositories: |
| 20 | + |
| 21 | +- [🤗Wan2.2-Fun-Inp-14B](https://huggingface.co/alibaba-pai/Wan2.2-Fun-A14B-InP) |
| 22 | +- Code repository: [VideoX-Fun](https://github.com/aigc-apps/VideoX-Fun) |
| 23 | + |
| 24 | +<UpdateReminder/> |
| 25 | + |
| 26 | +## Wan2.2 Fun Inp Start-End Frame Video Generation Workflow Example |
| 27 | + |
| 28 | +This workflow provides two versions: |
| 29 | +1. A version using [Wan2.2-Lightning](https://huggingface.co/lightx2v/Wan2.2-Lightning) 4-step LoRA from lightx2v for accelerated video generation |
| 30 | +2. A fp8_scaled version without acceleration LoRA |
| 31 | + |
| 32 | +Below are the test results using an RTX4090D 24GB VRAM GPU |
| 33 | + |
| 34 | +| Model Type | Resolution | VRAM Usage | First Generation Time | Second Generation Time | |
| 35 | +| ------------------------ | ---------- | ---------- | -------------------- | --------------------- | |
| 36 | +| fp8_scaled | 640×640 | 83% | ≈ 524s | ≈ 520s | |
| 37 | +| fp8_scaled + 4-step LoRA | 640×640 | 89% | ≈ 138s | ≈ 79s | |
| 38 | + |
| 39 | +Since the acceleration with LoRA is significant, the provided workflows enable the accelerated LoRA version by default. If you want to enable the other workflow, select it and use **Ctrl+B** to activate. |
| 40 | + |
| 41 | +### 1. Download Workflow File |
| 42 | + |
| 43 | +Please update your ComfyUI to the latest version, and find "**Wan2.2 Fun Inp**" under the menu `Workflow` -> `Browse Templates` -> `Video` to load the workflow. |
| 44 | + |
| 45 | +Or, after updating ComfyUI to the latest version, download the workflow below and drag it into ComfyUI to load. |
| 46 | + |
| 47 | +<video |
| 48 | + controls |
| 49 | + className="w-full aspect-video" |
| 50 | + src="https://raw.githubusercontent.com/Comfy-Org/example_workflows/refs/heads/main/video/wan/wan2.2_fun_inp/wan2.2_14B_fun_inp.mp4" |
| 51 | +></video> |
| 52 | + |
| 53 | +<a className="prose" target='_blank' href="https://raw.githubusercontent.com/Comfy-Org/workflow_templates/refs/heads/main/templates/video_wan2_2_14B_fun_inpaint.json" style={{ display: 'inline-block', backgroundColor: '#0078D6', color: '#ffffff', padding: '10px 20px', borderRadius: '8px', borderColor: "transparent", textDecoration: 'none', fontWeight: 'bold'}}> |
| 54 | + <p className="prose" style={{ margin: 0, fontSize: "0.8rem" }}>Download JSON Workflow</p> |
| 55 | +</a> |
| 56 | + |
| 57 | +Use the following materials as the start and end frames |
| 58 | + |
| 59 | + |
| 60 | + |
| 61 | + |
| 62 | +### 2. Manually Download Models |
| 63 | + |
| 64 | +**Diffusion Model** |
| 65 | +- [wan2.2_fun_inpaint_high_noise_14B_fp8_scaled.safetensors](https://huggingface.co/Comfy-Org/Wan_2.2_ComfyUI_Repackaged/resolve/main/split_files/diffusion_models/wan2.2_fun_inpaint_high_noise_14B_fp8_scaled.safetensors) |
| 66 | +- [wan2.2_fun_inpaint_low_noise_14B_fp8_scaled.safetensors](https://huggingface.co/Comfy-Org/Wan_2.2_ComfyUI_Repackaged/resolve/main/split_files/diffusion_models/wan2.2_fun_inpaint_low_noise_14B_fp8_scaled.safetensors) |
| 67 | + |
| 68 | +**Lightning LoRA (Optional, for acceleration)** |
| 69 | +- [wan2.2_i2v_lightx2v_4steps_lora_v1_high_noise.safetensors](https://huggingface.co/Comfy-Org/Wan_2.2_ComfyUI_Repackaged/resolve/main/split_files/loras/wan2.2_i2v_lightx2v_4steps_lora_v1_high_noise.safetensors) |
| 70 | +- [wan2.2_i2v_lightx2v_4steps_lora_v1_low_noise.safetensors](https://huggingface.co/Comfy-Org/Wan_2.2_ComfyUI_Repackaged/resolve/main/split_files/loras/wan2.2_i2v_lightx2v_4steps_lora_v1_low_noise.safetensors) |
| 71 | + |
| 72 | +**VAE** |
| 73 | +- [wan_2.1_vae.safetensors](https://huggingface.co/Comfy-Org/Wan_2.2_ComfyUI_Repackaged/resolve/main/split_files/vae/wan_2.1_vae.safetensors) |
| 74 | + |
| 75 | +**Text Encoder** |
| 76 | +- [umt5_xxl_fp8_e4m3fn_scaled.safetensors](https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/resolve/main/split_files/text_encoders/umt5_xxl_fp8_e4m3fn_scaled.safetensors) |
| 77 | + |
| 78 | +``` |
| 79 | +ComfyUI/ |
| 80 | +├───📂 models/ |
| 81 | +│ ├───📂 diffusion_models/ |
| 82 | +│ │ ├─── wan2.2_fun_inpaint_high_noise_14B_fp8_scaled.safetensors |
| 83 | +│ │ └─── wan2.2_fun_inpaint_low_noise_14B_fp8_scaled.safetensors |
| 84 | +│ ├───📂 loras/ |
| 85 | +│ │ ├─── wan2.2_i2v_lightx2v_4steps_lora_v1_high_noise.safetensors |
| 86 | +│ │ └─── wan2.2_i2v_lightx2v_4steps_lora_v1_low_noise.safetensors |
| 87 | +│ ├───📂 text_encoders/ |
| 88 | +│ │ └─── umt5_xxl_fp8_e4m3fn_scaled.safetensors |
| 89 | +│ └───📂 vae/ |
| 90 | +│ └── wan_2.1_vae.safetensors |
| 91 | +``` |
| 92 | + |
| 93 | +### 3. Step-by-Step Workflow Guide |
| 94 | + |
| 95 | + |
| 96 | + |
| 97 | +<Note> |
| 98 | + This workflow uses LoRA. Please make sure the corresponding Diffusion model and LoRA are matched. |
| 99 | +</Note> |
| 100 | + |
| 101 | +1. **High noise** model and **LoRA** loading |
| 102 | + - Ensure the `Load Diffusion Model` node loads the `wan2.2_fun_inpaint_high_noise_14B_fp8_scaled.safetensors` model |
| 103 | + - Ensure the `LoraLoaderModelOnly` node loads the `wan2.2_i2v_lightx2v_4steps_lora_v1_high_noise.safetensors` |
| 104 | +2. **Low noise** model and **LoRA** loading |
| 105 | + - Ensure the `Load Diffusion Model` node loads the `wan2.2_fun_inpaint_low_noise_14B_fp8_scaled.safetensors` model |
| 106 | + - Ensure the `LoraLoaderModelOnly` node loads the `wan2.2_i2v_lightx2v_4steps_lora_v1_low_noise.safetensors` |
| 107 | +3. Ensure the `Load CLIP` node loads the `umt5_xxl_fp8_e4m3fn_scaled.safetensors` model |
| 108 | +4. Ensure the `Load VAE` node loads the `wan_2.1_vae.safetensors` model |
| 109 | +5. Upload the start and end frame images as materials |
| 110 | +6. Enter your prompt in the Prompt group |
| 111 | +7. Adjust the size and video length in the `WanFunInpaintToVideo` node |
| 112 | + - Adjust the `width` and `height` parameters. The default is `640`. We set a smaller size, but you can modify it as needed. |
| 113 | + - Adjust the `length`, which is the total number of frames. The current workflow fps is 16. For example, if you want to generate a 5-second video, you should set it to 5*16 = 80. |
| 114 | +8. Click the `Run` button, or use the shortcut `Ctrl(cmd) + Enter` to execute video generation |
0 commit comments