请问有提供 WAN2.2 I2V 的微调代码示例吗？

[sd0809]

[bubbliiiing]

有的呀，有readme的https://github.com/aigc-apps/VideoX-Fun/blob/main/scripts/wan2.2/README_TRAIN.md

[bubbliiiing]

还是需要更精细的

[sd0809]

感谢您的回复！

在微调 wan2.2 I2V 的过程中我遇到了一个新的问题，是关于 float32 和 bf16 的。

https://github.com/aigc-apps/VideoX-Fun/blob/main/videox_fun/models/wan_transformer3d.py 的第 869 行强制使得 e0 是 float32 的，导致第 452 行 x = x + y * e[2] 后的x.dtype 为 float32, 导致了 457 行 self.norm3(x) 报错 RuntimeError: expected scalar type Float but found BFloat16。

请问这里为什么强制计算 e0 为 float32 呢？ 我能否直接把它转成 bf16 进行后续训练？

[sd0809]

class WanLayerNorm(nn.LayerNorm):

def __init__(self, dim, eps=1e-6, elementwise_affine=False):
    super().__init__(dim, elementwise_affine=elementwise_affine, eps=eps)

def forward(self, x):
    r"""
    Args:
        x(Tensor): Shape [B, L, C]
    """
    origin_dtype = x.dtype
    return F.layer_norm(
        x.float(),
        self.normalized_shape,
        self.weight.float() if self.weight is not None else None,
        self.bias.float() if self.bias is not None else None,
        self.eps,
    ).to(origin_dtype)

我仿照 diffusers 对 wan 的实现进行了修改，可以正常训练 bf16

[bubbliiiing]

能否把报错时的配置给我一下，我没遇到报错，我看看如何复现。

[bubbliiiing]

有可能是不同的accelerate配置导致的，我统一解决一下子

[sd0809]

{
"name": "train_wan2.2_lora",
"type": "debugpy",
"request": "launch",
"console": "integratedTerminal",
"justMyCode": false,
"module": "accelerate.commands.launch",
"env": {
"NCCL_DEBUG": "INFO",
"CUDA_VISIBLE_DEVICES": "0"
},
"args": [
"--use_deepspeed",
"--deepspeed_config_file", "configs/zero_stage2_config.json",
"--deepspeed_multinode_launcher", "standard",
"train_wan_2_2_lora.py",
"--config_path", "configs/wan_civitai_i2v.yaml",
"--pretrained_model_name_or_path", "Wan-AI/Wan2.2-I2V-A14B",
"--train_data_dir", "datasets/a/",
"--train_data_meta", "datasets/a.json",
"--image_sample_size", "720",
"--video_sample_size", "720",
"--token_sample_size", "720",
"--video_sample_stride", "2",
"--video_sample_n_frames", "81",
"--train_batch_size", "1",
"--video_repeat", "1",
"--gradient_accumulation_steps", "1",
"--dataloader_num_workers", "0",
"--num_train_epochs", "100",
"--checkpointing_steps", "50",
"--learning_rate", "1e-04",
"--seed", "42",
"--output_dir", "output_dir",
"--gradient_checkpointing",
"--mixed_precision", "bf16",
"--adam_weight_decay", "3e-2",
"--adam_epsilon", "1e-10",
"--vae_mini_batch", "1",
"--max_grad_norm", "0.05",
"--random_hw_adapt",
"--training_with_video_token_length",
"--enable_bucket",
"--uniform_sampling",
"--boundary_type", "low",
"--rank", "64",
"--network_alpha", "32",
"--target_name", "q,k,v,ffn.0,ffn.2",
"--use_peft_lora",
"--train_mode", "i2v"
]
}

zero_stage2_config.json 如下:

{
"bf16": {
"enabled": true
},
"train_micro_batch_size_per_gpu": 1,
"train_batch_size": "auto",
"gradient_accumulation_steps": "auto",
"dump_state": true,
"zero_optimization": {
"stage": 2,
"overlap_comm": true,
"contiguous_gradients": true,
"sub_group_size": 1e9,
"reduce_bucket_size": 5e8
}
}

辛苦您看一下！

[bubbliiiing]

看起来就是使用了zero2，很奇怪，我测测看

[bubbliiiing]

我使用如下代码似乎一切正常。太奇怪了。请问你是最新代码吗？

export MODEL_NAME="models/Diffusion_Transformer/Wan2.2-I2V-A14B"
export DATASET_NAME=""
export DATASET_META_NAME="datasets/demo/demo.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA. 
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO

accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_config.json --deepspeed_multinode_launcher standard scripts/wan2.2/train_lora.py \
  --config_path="config/wan2.2/wan_civitai_i2v.yaml" \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --train_data_dir=$DATASET_NAME \
  --train_data_meta=$DATASET_META_NAME \
  --image_sample_size=256 \
  --video_sample_size=256 \
  --token_sample_size=256 \
  --video_sample_stride=2 \
  --video_sample_n_frames=49 \
  --train_batch_size=1 \
  --video_repeat=1 \
  --gradient_accumulation_steps=1 \
  --dataloader_num_workers=8 \
  --num_train_epochs=500 \
  --checkpointing_steps=100 \
  --learning_rate=1e-04 \
  --seed=42 \
  --output_dir="output_dir" \
  --gradient_checkpointing \
  --mixed_precision="bf16" \
  --adam_weight_decay=3e-2 \
  --adam_epsilon=1e-10 \
  --vae_mini_batch=1 \
  --max_grad_norm=0.05 \
  --random_hw_adapt \
  --training_with_video_token_length \
  --enable_bucket \
  --uniform_sampling \
  --boundary_type="low" \
  --rank=128 \
  --network_alpha=128 \
  --target_name="q,k,v,ffn.0,ffn.2" \
  --use_peft_lora \
  --train_mode="i2v" \
  --low_vram

[sd0809]

是最新版的....

我再重新试一下吧，感谢