How to run the inference code?

[YIYANGCAI]

Thanks for your excellent work!

I have a question of reference-to-video inference. For the file ./examples/wanvideo/model_inference/Wan2.1-T2V-14B_subject.py. Where should I place the path of download Wan2.1-T2V-14B and RefAlign-14B in this file to ensure the proper inference process?

I am looking forward to your response!

[gudaochangsheng]

Thank you for your interest in RefAlign.

RefAlign/examples/wanvideo/model_inference/Wan2.1-T2V-14B_subject.py

Line 87 in 2f6beac

"model.safetensors",

Using RefAlign-14B (DiT weights)

---

RefAlign/examples/wanvideo/model_inference/Wan2.1-T2V-14B_subject.py

Line 89 in 2f6beac

ModelConfig(path=".../DiffSynthStudio/models/train/wan14b/models_t5_umt5-xxl-enc-bf16.pth"),

RefAlign/examples/wanvideo/model_inference/Wan2.1-T2V-14B_subject.py

Line 90 in 2f6beac

ModelConfig(path=".../DiffSynthStudio/models/train/wan14b/Wan2.1_VAE.pth")],

Using Wan2.1-14B (T5 and VAE weights)

[YIYANGCAI]

Thanks for your quick response!

I have follow your hints and used weights from local directory as:

pipe = WanVideoPipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda:4",
    # model_configs=[
    #     ModelConfig(model_id="Wan-AI/Wan2.1-T2V-1.3B", origin_file_pattern="diffusion_pytorch_model*.safetensors", offload_device="cpu"),
    #     ModelConfig(model_id="Wan-AI/Wan2.1-T2V-1.3B", origin_file_pattern="models_t5_umt5-xxl-enc-bf16.pth", offload_device="cpu"),
    #     ModelConfig(model_id="Wan-AI/Wan2.1-T2V-1.3B", origin_file_pattern="Wan2.1_VAE.pth", offload_device="cpu"),
    # ],
    model_configs=[
    ModelConfig(model_id="/mnt/models/hf_model/RefAlign-14B", origin_file_pattern="model*.safetensors", offload_device="cpu"),
    ModelConfig(path="/mnt/models/hf_model/Wan2.1-T2V-14B/models_t5_umt5-xxl-enc-bf16.pth", offload_device="cpu"),
    ModelConfig(path="/mnt/models/hf_model/Wan2.1-T2V-14B/Wan2.1_VAE.pth", offload_device="cpu")],
)

Does such action correct? Since it returns error:

huggingface_hub.errors.HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: '[]'.

when loading the tokenizer. I would appreciate it if you can give me more hints. Thanks you!

[gudaochangsheng]

Hi, thanks for your interest!

Our model configuration is the same as follows:

model_configs=[
    ModelConfig(path=[
        ".../RefAlign-14B/model-00000.safetensors",
        ".../RefAlign-14B/model-00001.safetensors",
        ".../RefAlign-14B/model-00002.safetensors",
        ".../RefAlign-14B/model-00003.safetensors",
        ".../RefAlign-14B/model-00004.safetensors",
        ".../RefAlign-14B/model-00005.safetensors",
    ]),
    ModelConfig(path=".../models/train/wan14b/models_t5_umt5-xxl-enc-bf16.pth"),
    ModelConfig(path=".../models/train/wan14b/Wan2.1_VAE.pth")
]