Exception: Error during training: Given groups=1, weight of size [3072, 16, 1, 2, 2], expected input[1, 8, 1, 56, 68] to have 16 channels, but got 8 channels instead

[neph1]

System Info / 系統信息

I've just tried out the v0.1.0 with both ptd and accelerate. Both throw exceptions after about an epoch (62 samples).

Training steps: 21%|██▏ | 64/300 [04:09<07:50, 1.99s/it, grad_norm=0.00579, global_avg_loss=0.338, global_max_loss=0.338]
Training steps: 22%|██▏ | 65/300 [04:11<07:46, 1.98s/it, grad_norm=0.00579, global_avg_loss=0.338, global_max_loss=0.338]
Training steps: 22%|██▏ | 65/300 [04:11<07:46, 1.98s/it] 2025-03-24 20:50:21,282 - finetrainers - ERROR - Error during training: Given groups=1, weight of size [3072, 16, 1, 2, 2], expected input[1, 8, 1, 56, 68] to have 16 channels, but got 8 channels instead
2025-03-24 20:50:21,283 - finetrainers - ERROR - An error occurred during training: Given groups=1, weight of size [3072, 16, 1, 2, 2], expected input[1, 8, 1, 56, 68] to have 16 channels, but got 8 channels instead
2025-03-24 20:50:21,292 - finetrainers - ERROR - Traceback (most recent call last):
File "train.py", line 70, in main
trainer.run()
File "finetrainers/trainer/sft_trainer/trainer.py", line 97, in run
raise e
File "finetrainers/trainer/sft_trainer/trainer.py", line 92, in run
self._train()
File "finetrainers/trainer/sft_trainer/trainer.py", line 473, in _train
pred, target, sigmas = self.model_specification.forward(
File "finetrainers/models/hunyuan_video/base_specification.py", line 317, in forward
pred = transformer(
File "/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/python3.10/site-packages/accelerate/utils/operations.py", line 819, in forward
return model_forward(*args, **kwargs)
File "/python3.10/site-packages/accelerate/utils/operations.py", line 807, in call
return convert_to_fp32(self.model_forward(*args, **kwargs))
File "/python3.10/site-packages/torch/amp/autocast_mode.py", line 44, in decorate_autocast
return func(*args, **kwargs)
File "/diffusers/src/diffusers/models/transformers/transformer_hunyuan_video.py", line 720, in forward
hidden_states = self.x_embedder(hidden_states)
File "/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/diffusers/src/diffusers/models/transformers/transformer_hunyuan_video.py", line 154, in forward
hidden_states = self.proj(hidden_states)
File "/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/diffusers/src/diffusers/hooks/hooks.py", line 148, in new_forward
output = function_reference.forward(*args, **kwargs)
File "/python3.10/site-packages/torch/nn/modules/conv.py", line 725, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/python3.10/site-packages/torch/nn/modules/conv.py", line 720, in _conv_forward
return F.conv3d(
RuntimeError: Given groups=1, weight of size [3072, 16, 1, 2, 2], expected input[1, 8, 1, 56, 68] to have 16 channels, but got 8 channels instead

I'm currently on diffusers commit e7e6d852822b279b88f133395bcc2dd056eb59da
v0.0.1 still works
Let me know if this is a configuration issue
I'm using the ui, but you can see the command it generates below.

Information / 问题信息

• The official example scripts / 官方的示例脚本
• My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程

accelerate launch --config_file /accelerate_configs/uncompiled_1.yaml --gpu_ids 0 /train.py --parallel_backend accelerate --pp_degree 1 --dp_degree 1 --dp_shards 1 --cp_degree 1 --tp_degree 1 --model_name hunyuan_video --pretrained_model_name_or_path models/hunyuan --text_encoder_dtype bf16 --text_encoder_2_dtype bf16 --text_encoder_3_dtype bf16 --transformer_dtype bf16 --vae_dtype bf16 --layerwise_upcasting_modules transformer --layerwise_upcasting_storage_dtype float8_e4m3fn --layerwise_upcasting_skip_modules_pattern patch_embed pos_embed x_embedder context_embedder ^proj_in$ ^proj_out$ norm --dataset_config /dataset_config.json --caption_dropout_p 0.05 --caption_dropout_technique empty --enable_precomputation --precomputation_items 62 --precomputation_once --dataloader_num_workers 0 --training_type lora --seed 425 --batch_size 1 --train_steps 300 --rank 64 --lora_alpha 64 --target_modules to_q to_k to_v to_out.0 --gradient_accumulation_steps 8 --gradient_checkpointing --checkpointing_steps 100 --checkpointing_limit 5 --enable_slicing --enable_tiling --optimizer adamw --lr 0.0002 --lr_scheduler linear --lr_warmup_steps 100 --lr_num_cycles 1 --beta1 0.9 --beta2 0.95 --weight_decay 0.001 --epsilon 1e-8 --max_grad_norm 1 --num_validation_videos 0 --validation_steps 10000 --tracker_name finetrainers --output_dir output/ --nccl_timeout 1800 --report_to none

torchrun --standalone --nnodes 1 --nproc_per_node 1 --rdzv_backend c10d --rdzv_endpoint localhost:0 /train.py --parallel_backend ptd --pp_degree 1 --dp_degree 1 --dp_shards 1 --cp_degree 1 --tp_degree 1 --model_name hunyuan_video --pretrained_model_name_or_path models/hunyuan --text_encoder_dtype bf16 --text_encoder_2_dtype bf16 --text_encoder_3_dtype bf16 --transformer_dtype bf16 --vae_dtype bf16 --layerwise_upcasting_modules transformer --layerwise_upcasting_storage_dtype float8_e4m3fn --layerwise_upcasting_skip_modules_pattern patch_embed pos_embed x_embedder context_embedder ^proj_in$ ^proj_out$ norm --dataset_config /dataset_config.json --caption_dropout_p 0.05 --caption_dropout_technique empty --enable_precomputation --precomputation_items 62 --precomputation_once --dataloader_num_workers 0 --training_type lora --seed 425 --batch_size 1 --train_steps 300 --rank 64 --lora_alpha 64 --target_modules to_q to_k to_v to_out.0 --gradient_accumulation_steps 8 --gradient_checkpointing --checkpointing_steps 100 --checkpointing_limit 5 --enable_slicing --enable_tiling --optimizer adamw --lr 0.0002 --lr_scheduler linear --lr_warmup_steps 100 --lr_num_cycles 1 --beta1 0.9 --beta2 0.95 --weight_decay 0.001 --epsilon 1e-8 --max_grad_norm 1 --num_validation_videos 0 --validation_steps 10000 --tracker_name finetrainers --output_dir output/ --nccl_timeout 1800 --report_to none

Expected behavior / 期待表现

.

[neph1]

To add some information, when I tested this about a week ago or so, I ran tests that were 65 steps long. The resulting checkpoint had been deleted, though. Maybe a coincidence, but you never know.

[neph1]

Ah. The "precomputed" folder had items from previous runs, and I guess they somehow interfered. Maybe because it had mixed ptd and accelerate items.
When I deleted the folder training continued past the 65 steps mark. This can be closed, but I'll leave it open if you want to take some action.
Currently running accelerate.

[a-r-r-o-w]

Ohh, this should ideally never happen 😬🤔 There is code in place to make sure previous data is deleted (and I've verified it to work 100% of the time in the past) so that this does not happen since I've run into same issue in past.

Will have to investigate why it's not working the way I have it in mind. I'm vacationing with the diffusers team this week so will be a bit slow, but will try to fix behaviour as soon as i can

[neph1]

When trying again, it might be related to precomputate_once. I got the issue again (and had not changed backends), several times. But dropping the setting made it work.

[a-r-r-o-w]

I couldn't look into it yet, but am back from vacationing and will get to testing and fixing these behaviours. I believe it should be related to folders not being cleaned up somewhere, but could be something else entirely

[Eurus-Holmes]

@neph1 I got the same error with you, I was trying to replace Hunyuan's transformer with Framepack's transformer to train lora, any advice?

[a-r-r-o-w]

@neph1 @Eurus-Holmes Could you update to the latest main branch and see if you get the same error? I made a change in #384 that should be correct based on what I initially intented to do with precomputation_once. For a quick test, I would try with:

--enable_precomputation
--precomputation_items 1
--precomputation_once

[Eurus-Holmes]

@a-r-r-o-w I pulled the latest main branch but still got the same error, how about you? @neph1

[Eurus-Holmes]

@a-r-r-o-w I pulled the latest main branch but still got the same error, how about you? @neph1

Ok when I removed my old "precomputed" folder, it's working but got another error ValueError: Dataset does not support precomputing all embeddings at once. I need to change MAX_PRECOMPUTABLE_ITEMS_LIMIT

[a-r-r-o-w]

Yeah the new error makes sense if you're using a large enough dataset. The limit should really be removed eventually once there's support for re-using precomputef embeds. The precomputation folder should also really be deleted by itself in this case.

I'll try to see why that doesn't seem to be happening. Thanks for testing!

[Eurus-Holmes]

Yeah the new error makes sense if you're using a large enough dataset. The limit should really be removed eventually once there's support for re-using precomputef embeds. The precomputation folder should also really be deleted by itself in this case.

I'll try to see why that doesn't seem to be happening. Thanks for testing!

Also when set precomputation_items=1, it should be training slower right? Since it's kind of computing features on the fly when training, but I found it seems very fast and only saved a few samples in precomputed dir not all samples been processed

[a-r-r-o-w]

If you start with a clean directory and set precomputation_items=1 on a single GPU, it should save a single condition and latent .pt file. Training might seem faster because instead of computing text embeddings and latents at each step of training, it is using the data already available on the disk.

Depending on what kind of disk you have and amount of CPU RAM, training speed may vary, but if you have a decent one, I would recommend using --precomputation_items <size_of_dataset> --precomputation_once (without --enable_precomputation). This will keep the text embeddings and latents in the RAM instead of disk and enable even faster training (as a result RAM usage will be higher. For this reason, the hardcoded limit is 1024 items)

[Eurus-Holmes]

If you start with a clean directory and set precomputation_items=1 on a single GPU, it should save a single condition and latent .pt file. Training might seem faster because instead of computing text embeddings and latents at each step of training, it is using the data already available on the disk.

Depending on what kind of disk you have and amount of CPU RAM, training speed may vary, but if you have a decent one, I would recommend using --precomputation_items <size_of_dataset> --precomputation_once (without --enable_precomputation). This will keep the text embeddings and latents in the RAM instead of disk and enable even faster training (as a result RAM usage will be higher. For this reason, the hardcoded limit is 1024 items)

Ok got it, thank you! So when set precomputation_items=1, it will just override the processed samples, so it will always only one sample on a single GPU, right? I think that's make sense, but my question is compare to set precomputation_items=1, if set --precomputation_items <size_of_dataset> it will take a very very long time to process like a few days (about 10k video clips each 10s), is that expected?

[a-r-r-o-w]

Yes 10k clips will take a very long time with current setup. And the precomputed latents/conditions are not re-usable across different runs at the moment. This is because, under my initial assumptions when this was first implemented, lora training for concepts/styles/character is usually done with < 50-100 images/videos. At that scale, repeating precomputation for different runs isn't usually a problem IMO, but I can see why that's problematic as soon as you start increasing dataset size even a little bit.

There are two things I have in mind to improve this experience but haven't yet found the time to implement:

• Support loading existing precomputation directories, so that they can be re-used across different runs (1)
• Support precomputation separately from the training, so that if users want to, they can do distributed precomputation first, analyze their data, filter and remove things they don't want, or just perform any other necessary preprocessing on the precomputed data. (2)

My current plans for scaling training to 1M+ scale is to do it without precomputation because storage/loading may actually be slower/costlier compared to applying different parallelisms to the text encoder and vae (FSDP/tensor/context parallel) and doing on-the-fly computation of required embeddings. Due to this reason, I've been slower on optimizing for other degrees of freedom.

(1) will be quickly implementable. I'll open a PR for this today

[Eurus-Holmes]

Yes 10k clips will take a very long time with current setup. And the precomputed latents/conditions are not re-usable across different runs at the moment. This is because, under my initial assumptions when this was first implemented, lora training for concepts/styles/character is usually done with < 50-100 images/videos. At that scale, repeating precomputation for different runs isn't usually a problem IMO, but I can see why that's problematic as soon as you start increasing dataset size even a little bit.

There are two things I have in mind to improve this experience but haven't yet found the time to implement:

• Support loading existing precomputation directories, so that they can be re-used across different runs (1)
• Support precomputation separately from the training, so that if users want to, they can do distributed precomputation first, analyze their data, filter and remove things they don't want, or just perform any other necessary preprocessing on the precomputed data. (2)

My current plans for scaling training to 1M+ scale is to do it without precomputation because storage/loading may actually be slower/costlier compared to applying different parallelisms to the text encoder and vae (FSDP/tensor/context parallel) and doing on-the-fly computation of required embeddings. Due to this reason, I've been slower on optimizing for other degrees of freedom.

(1) will be quickly implementable. I'll open a PR for this today

yeah both make sense to me, appreciate it!