[Feature] Development Roadmap (V1 Training & Distillation & VSA Release)

[jzhang38]

Contritbutions are welcome!

Focus

• Efficient video model finetuning (T2V and I2V)
• Distillation (DMD & PCM)
• VSA Kernel Release
• Data loader optimization
• Mixed-resolution and no-precompute support (T5, VAE, LoRA)

Model Development & Finetuning

• T2V finetuning experiments on crush-smol @SolitaryThinker @jzhang38
• I2V finetuning experiments on crush-smol @JerryZhou54 @SolitaryThinker [Feature] [Training] Add i2v training #559
• Fine-tune VSA-based T2V model @BrianChen1129 [Feature] Adding VSA inference #478

Distillation & Alignment

• V1 PCM Distillation @jzhang38 @Eigensystem PCM Distillation #471
• V1 DMD Distillation @jzhang38 @Eigensystem

Preprocessing & Dataset

• Refactor map-style parquet dataset @jzhang38 [Feat][Dataloader] 1/n Refactor parquet map-style dataloader #492
• Write iterable, resumable style parquet dataset @jzhang38 [Feat][Dataloader] 1/n Refactor parquet map-style dataloader #492
• Refactor preprocess @SolitaryThinker [Training] Refactor and improve validation datasets #539
• Latent precomputation for image-to-video training @BrianChen1129
• benchmark dataloader speed, implement iterable dataset and @SolitaryThinker
• Multi-resolution bucketing for long video input compatibility @rlsu9 @Eigensystem

Training Infrastructure

• Enable checkpoint saving/resume for distributed experiments @kevin314
• LoRA Inference @Edenzzzz [LoRA] Support V1 LoRA inference #451
• Accelerate weight loading [Feature] Load weights from distributed #470
• LoRA Training @Edenzzzz [LoRA] Support v1 LoRA training  #576
• Support Video-Fun Wan 1.3B I2V model @SolitaryThinker @JerryZhou54
• Activation offloading @JerryZhou54
• Various Activation checkpointing methods @Eigensystem [Feat] activation checkpointing #584
• Torch compile and regional compile @jzhang38 [Feature] Optionally enable torch compile  #684
• LigerKernels/SGLang cuda kernels for norm & activation dispatch: Call out for contributions!
• Enable training with no precomputed latents @rlsu9 @Eigensystem
• Reduce the number of changed layers in _param_names_mapping @jzhang38

VSA Kernel Release

• VSA H100 Kernel @jzhang38 [Feature] Adding VSA inference #478
• VSA 4090 Kernel @jzhang38

CI Infrastructure

• Migrate to BuildKite + Modal

Related resources

No response

[SolitaryThinker]

For future reference related to dataloader and dataset refactors: https://github.com/hao-ai-lab/FastVideo/pull/459/files#r2124823857

[philippe-eecs]

Do you know when you will finish the v1 finetune pipeline for wan? I currently am getting lots of weird shape errors and bugs with I try running something with 8 GPUs. I think the code hasn't been totally finish right?

[SolitaryThinker]

Hi @philippe-eecs , we just merged #459 which included many fixes for training. Please see if that resolves your issue? Do note that if you are using wan 1.3B, you cannot use sp=8 as it only has 12 attention heads. However it will work with dp=2 sp=4

[philippe-eecs]

Yes I just saw your merged PR. I will check it out soon thanks! I also noticed I setup my dataset wrong and the width was >832 for some of my videos my bad...

As for DP=2, SP=4, TP=4 I don't follow what the parallelisms are doing very well from your codebase. I saw that SP == TP == NUM GPUS? Is that different now? Just wondering do you have more documentation on how you setup the parallelism (and potentially some learning resources for this I am very new to parallelism strategies)