Skip to content

Commit f166e7e

Browse files
SamYuan1990stevhliu
authored andcommitted
add two more docs
Signed-off-by: SamYuan1990 <[email protected]>
1 parent 5bfe472 commit f166e7e

File tree

2 files changed

+661
-0
lines changed

2 files changed

+661
-0
lines changed
Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# 将模型适配至新任务
2+
3+
许多扩散系统共享相同的组件架构,这使得您能够将针对某一任务预训练的模型调整适配至完全不同的新任务。
4+
5+
本指南将展示如何通过初始化并修改预训练 [`UNet2DConditionModel`] 的架构,将文生图预训练模型改造为图像修复(inpainting)模型。
6+
7+
## 配置 UNet2DConditionModel 参数
8+
9+
默认情况下,[`UNet2DConditionModel`][输入样本](https://huggingface.co/docs/diffusers/v0.16.0/en/api/models#diffusers.UNet2DConditionModel.in_channels)接受4个通道。例如加载 [`stable-diffusion-v1-5/stable-diffusion-v1-5`](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5) 这样的文生图预训练模型,查看其 `in_channels` 参数值:
10+
11+
```python
12+
from diffusers import StableDiffusionPipeline
13+
14+
pipeline = StableDiffusionPipeline.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5", use_safetensors=True)
15+
pipeline.unet.config["in_channels"]
16+
4
17+
```
18+
19+
而图像修复任务需要输入样本具有9个通道。您可以在 [`runwayml/stable-diffusion-inpainting`](https://huggingface.co/runwayml/stable-diffusion-inpainting) 这样的预训练修复模型中验证此参数:
20+
21+
```python
22+
from diffusers import StableDiffusionPipeline
23+
24+
pipeline = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-inpainting", use_safetensors=True)
25+
pipeline.unet.config["in_channels"]
26+
9
27+
```
28+
29+
要将文生图模型改造为修复模型,您需要将 `in_channels` 参数从4调整为9。
30+
31+
初始化一个加载了文生图预训练权重的 [`UNet2DConditionModel`],并将 `in_channels` 设为9。由于输入通道数变化导致张量形状改变,需要设置 `ignore_mismatched_sizes=True``low_cpu_mem_usage=False` 来避免尺寸不匹配错误。
32+
33+
```python
34+
from diffusers import AutoModel
35+
36+
model_id = "stable-diffusion-v1-5/stable-diffusion-v1-5"
37+
unet = AutoModel.from_pretrained(
38+
model_id,
39+
subfolder="unet",
40+
in_channels=9,
41+
low_cpu_mem_usage=False,
42+
ignore_mismatched_sizes=True,
43+
use_safetensors=True,
44+
)
45+
```
46+
47+
此时文生图模型的其他组件权重仍保持预训练状态,但UNet的输入卷积层权重(`conv_in.weight`)会随机初始化。由于这一关键变化,必须对模型进行修复任务的微调,否则模型将仅会输出噪声。
48+
Disclaimers: This content is powered by i18n-agent-action with LLM service https://api.deepseek.com with model deepseek-chat, for some reason, (for example, we are not native speaker) we use LLM to provide this translate for you. If you find any corrections, please file an issue or raise a PR back to github, and switch back to default language.

0 commit comments

Comments
 (0)