none get_lr_scheduler_state()

[ootsuka-repos]

System Info / 系統信息

https://github.com/a-r-r-o-w/finetrainers/tree/226c98d39c55cc1b377035d9a7c629393455deaf

Information / 问题信息

• The official example scripts / 官方的示例脚本
• My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程

2025-03-14 14:56:17,267 - finetrainers - DEBUG - AccelerateParallelBackend::prepare_dataloader completed!
2025-03-14 14:56:17,267 - finetrainers - ERROR - Error during training: 'AcceleratedScheduler' object has no attribute 'get_lr_scheduler_state'
2025-03-14 14:56:17,547 - finetrainers - ERROR - An error occurred during training: 'AcceleratedScheduler' object has no attribute 'get_lr_scheduler_state'
2025-03-14 14:56:17,548 - finetrainers - ERROR - Traceback (most recent call last):
File "/home/user/デスクトップ/finetrainers/train.py", line 70, in main
trainer.run()
File "/home/user/デスクトップ/finetrainers/finetrainers/trainer/sft_trainer/trainer.py", line 97, in run
raise e
File "/home/user/デスクトップ/finetrainers/finetrainers/trainer/sft_trainer/trainer.py", line 91, in run
self._prepare_checkpointing()
File "/home/user/デスクトップ/finetrainers/finetrainers/trainer/sft_trainer/trainer.py", line 312, in _prepare_checkpointing
self.checkpointer = utils.PTDCheckpointManager(
File "/home/user/デスクトップ/finetrainers/finetrainers/utils/state_checkpoint.py", line 65, in init
self.states.update(schedulers.get_lr_scheduler_state())
AttributeError: 'AcceleratedScheduler' object has no attribute 'get_lr_scheduler_state'

wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync logs/wandb/offline-run-20250314_145616-ocyb0au2

• echo -ne '-------------------- Finished executing script --------------------\n\n'
  -------------------- Finished executing script --------------------

Expected behavior / 期待表现

存在すること。動作すること。

[tolgacangoz]

Hi. Duplicate #293?

[ootsuka-repos]

That's a duplicate, thanks!

[tolgacangoz]

Hey, I wonder if torchrun works?

[ootsuka-repos]

I get an error when I run it with torchrun.
I may have to wait a while.

[tolgacangoz]

Thanks for the update. Could you share the reproducible command, environment, full error trace, etc.?

[ootsuka-repos]

(.venv) (base) user@user-Super-Server:~/デスクトップ/finetrainers$ ./train.sh

• export WANDB_MODE=offline
• WANDB_MODE=offline
• export NCCL_P2P_DISABLE=1
• NCCL_P2P_DISABLE=1
• export TORCH_NCCL_ENABLE_MONITORING=0
• TORCH_NCCL_ENABLE_MONITORING=0
• export FINETRAINERS_LOG_LEVEL=DEBUG
• FINETRAINERS_LOG_LEVEL=DEBUG
• BACKEND=ptd
• NUM_GPUS=2
• CUDA_VISIBLE_DEVICES=0,1
• TRAINING_DATASET_CONFIG=training.json
• VALIDATION_DATASET_FILE=validation.json
• DDP_1='--parallel_backend ptd --pp_degree 1 --dp_degree 1 --dp_shards 1 --cp_degree 1 --tp_degree 1'
• DDP_2='--parallel_backend ptd --pp_degree 1 --dp_degree 2 --dp_shards 1 --cp_degree 1 --tp_degree 1'
• DDP_4='--parallel_backend ptd --pp_degree 1 --dp_degree 4 --dp_shards 1 --cp_degree 1 --tp_degree 1'
• FSDP_2='--parallel_backend ptd --pp_degree 1 --dp_degree 1 --dp_shards 2 --cp_degree 1 --tp_degree 1'
• FSDP_4='--parallel_backend ptd --pp_degree 1 --dp_degree 1 --dp_shards 4 --cp_degree 1 --tp_degree 1'
• HSDP_2_2='--parallel_backend ptd --pp_degree 1 --dp_degree 2 --dp_shards 2 --cp_degree 1 --tp_degree 1'
• parallel_cmd=($DDP_2)
• model_cmd=(--model_name "wan" --pretrained_model_name_or_path "Wan-AI/Wan2.1-T2V-1.3B-Diffusers")
• dataset_cmd=(--dataset_config $TRAINING_DATASET_CONFIG --dataset_shuffle_buffer_size 10 --enable_precomputation --precomputation_items 20 --precomputation_once)
• dataloader_cmd=(--dataloader_num_workers 0)
• diffusion_cmd=(--flow_weighting_scheme "logit_normal")
• training_cmd=(--training_type "lora" --seed 42 --batch_size 1 --train_steps 30000 --rank 32 --lora_alpha 32 --target_modules "blocks.*(to_q|to_k|to_v|to_out.0)" --gradient_accumulation_steps 1 --gradient_checkpointing --checkpointing_steps 500 --checkpointing_limit 2 --enable_slicing --enable_tiling)
• optimizer_cmd=(--optimizer "adamw" --lr 5e-5 --lr_scheduler "constant_with_warmup" --lr_warmup_steps 1000 --lr_num_cycles 1 --beta1 0.9 --beta2 0.99 --weight_decay 1e-4 --epsilon 1e-8 --max_grad_norm 1.0)
• validation_cmd=(--validation_dataset_file "$VALIDATION_DATASET_FILE" --validation_steps 500000)
• miscellaneous_cmd=(--tracker_name "finetrainers-wan" --output_dir "wan" --init_timeout 600 --nccl_timeout 600 --report_to "wandb")
• '[' ptd == accelerate ']'
• '[' ptd == ptd ']'
• export CUDA_VISIBLE_DEVICES=0,1
• CUDA_VISIBLE_DEVICES=0,1
• torchrun --standalone --nnodes=1 --nproc_per_node=2 --rdzv_backend c10d --rdzv_endpoint=localhost:0 train.py --parallel_backend ptd --pp_degree 1 --dp_degree 2 --dp_shards 1 --cp_degree 1 --tp_degree 1 --model_name wan --pretrained_model_name_or_path Wan-AI/Wan2.1-T2V-1.3B-Diffusers --dataset_config training.json --dataset_shuffle_buffer_size 10 --enable_precomputation --precomputation_items 20 --precomputation_once --dataloader_num_workers 0 --flow_weighting_scheme logit_normal --training_type lora --seed 42 --batch_size 1 --train_steps 30000 --rank 32 --lora_alpha 32 --target_modules 'blocks.*(to_q|to_k|to_v|to_out.0)' --gradient_accumulation_steps 1 --gradient_checkpointing --checkpointing_steps 500 --checkpointing_limit 2 --enable_slicing --enable_tiling --optimizer adamw --lr 5e-5 --lr_scheduler constant_with_warmup --lr_warmup_steps 1000 --lr_num_cycles 1 --beta1 0.9 --beta2 0.99 --weight_decay 1e-4 --epsilon 1e-8 --max_grad_norm 1.0 --validation_dataset_file validation.json --validation_steps 500000 --tracker_name finetrainers-wan --output_dir wan --init_timeout 600 --nccl_timeout 600 --report_to wandb
  W0314 18:44:47.926000 111027 torch/distributed/run.py:792]
  W0314 18:44:47.926000 111027 torch/distributed/run.py:792] *****************************************
  W0314 18:44:47.926000 111027 torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
  W0314 18:44:47.926000 111027 torch/distributed/run.py:792] *****************************************
  2025-03-14 18:44:53,339 - finetrainers - DEBUG - Successfully imported bitsandbytes version 0.45.3
  2025-03-14 18:44:53,344 - finetrainers - DEBUG - Remaining unparsed arguments: []
  2025-03-14 18:44:55,509 - finetrainers - INFO - Initialized parallel state with:
  • World size: 2
  • Pipeline parallel degree: 1
  • Data parallel degree: 2
  • Context parallel degree: 1
  • Tensor parallel degree: 1
  • Data parallel shards: 1

2025-03-14 18:44:55,513 - finetrainers - DEBUG - Device mesh: DeviceMesh('cuda', [0, 1], mesh_dim_names=('dp_replicate',))
2025-03-14 18:44:55,513 - finetrainers - DEBUG - Enabling determinism: {'global_rank': 0, 'seed': 42}
2025-03-14 18:44:55,664 - finetrainers - INFO - Initializing models
Fetching 2 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 18275.83it/s]
Fetching 2 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 15709.00it/s]
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:02<00:00, 1.22s/it]
2025-03-14 18:44:58,776 - finetrainers - INFO - Initializing trainable parameters
2025-03-14 18:44:58,776 - finetrainers - INFO - Finetuning transformer with PEFT parameters
2025-03-14 18:44:59,459 - finetrainers - INFO - Applying DDP to the model
2025-03-14 18:44:59,485 - finetrainers - DEBUG - Applied PytorchDTensorParallel::apply_ddp to model.
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:02<00:00, 1.13s/it]
2025-03-14 18:45:00,419 - finetrainers - INFO - Initializing optimizer and lr scheduler
2025-03-14 18:45:00,426 - finetrainers - DEBUG - PytorchDTensorParallelBackend::prepare_optimizer completed!
2025-03-14 18:45:00,426 - finetrainers - INFO - Initialized FineTrainers
2025-03-14 18:45:00,427 - finetrainers - INFO - Initializing trackers: ['wandb']. Logging to log_dir='logs'
wandb: Using wandb-core as the SDK backend. Please refer to https://wandb.me/wandb-core for more information.
wandb: Tracking run with wandb version 0.19.8
wandb: W&B syncing is set to offline in this directory.
wandb: Run wandb online or set WANDB_MODE=online to enable cloud syncing.
2025-03-14 18:45:00,936 - finetrainers - INFO - WandB logging enabled
2025-03-14 18:45:00,936 - finetrainers - INFO - Initializing dataset and dataloader
2025-03-14 18:45:00,936 - finetrainers - INFO - Training configured to use 1 datasets
2025-03-14 18:45:00,944 - finetrainers - INFO - Initialized dataset: /home/user/デスクトップ/finetrainers/nsfw_anime
2025-03-14 18:45:00,944 - finetrainers - DEBUG - PytorchDTensorParallelBackend::prepare_dataset completed!
2025-03-14 18:45:00,944 - finetrainers - INFO - Initializing IterableDatasetPreprocessingWrapper for the dataset with the following configuration:

• Dataset Type: video
• ID Token: nsfw_anime
• Image Resolution Buckets: None
• Video Resolution Buckets: [[49, 512, 768], [49, 768, 512]]
• Reshape Mode: bicubic
• Remove Common LLM Caption Prefixes: True

2025-03-14 18:45:00,944 - finetrainers - INFO - Initializing IterableCombinedDataset with the following configuration:

• Number of Datasets: 1
• Buffer Size: 10
• Shuffle: True

2025-03-14 18:45:00,945 - finetrainers - DEBUG - PytorchDTensorParallelBackend::prepare_dataloader completed!
2025-03-14 18:45:00,945 - finetrainers - INFO - Checkpointing enabled. Checkpoints will be stored in 'wan'
2025-03-14 18:45:00,945 - finetrainers - INFO - Starting training
2025-03-14 18:45:00,946 - finetrainers - INFO - Memory before training start: {
"memory_allocated": 2.738,
"memory_reserved": 2.967,
"max_memory_allocated": 2.738,
"max_memory_reserved": 2.967
}
2025-03-14 18:45:00,946 - finetrainers - INFO - Training configuration: {
"trainable parameters": 23592960,
"train steps": 30000,
"per-replica batch size": 1,
"global batch size": 2,
"gradient accumulation steps": 1
}
Training steps: 0%| | 0/30000 [00:00<?, ?it/s]2025-03-14 18:45:00,957 - finetrainers - DEBUG - Deleting files: []
2025-03-14 18:45:00,957 - finetrainers - INFO - Precomputed condition & latent data exhausted. Loading & preprocessing new data.
Downloading shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 2177.73it/s]
Downloading shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 16487.04it/s]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:50<00:00, 10.11s/it]
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:50<00:00, 10.10s/it]
Processing data on rank 1: 0%| | 0/20 [00:00<?, ?it/s2025-03-14 18:46:29,643 - finetrainers - INFO - Starting IterableCombinedDataset with 1 datasets | 0/20 [00:00<?, ?it/s]
2025-03-14 18:46:29,644 - finetrainers - INFO - Starting IterableDatasetPreprocessingWrapper for the dataset
Filling buffer from data iterator 0: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:54<00:00, 5.50s/it]
Filling buffer from data iterator 0: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [01:37<00:00, 9.76s/it]
Processing data on rank 0: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [02:37<00:00, 7.88s/it]
Processing data on rank 0: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:36<00:00, 1.83s/it]
2025-03-14 18:49:55,574 - finetrainers - DEBUG - Starting training step (1/30000)████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:36<00:00, 1.81s/it]
Processing data on rank 1: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [04:43<00:00, 14.17s/it]
Processing data on rank 1: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:34<00:00, 1.75s/it]
2025-03-14 18:51:49,978 - finetrainers - ERROR - Error during training: mat1 and mat2 must have the same dtype, but got BFloat16 and Float
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync logs/wandb/offline-run-20250314_184500-18838pjb
wandb: Find logs at: logs/wandb/offline-run-20250314_184500-18838pjb/logs
2025-03-14 18:51:50,041 - finetrainers - ERROR - An error occurred during training: mat1 and mat2 must have the same dtype, but got BFloat16 and Float
2025-03-14 18:51:50,044 - finetrainers - ERROR - Traceback (most recent call last):
File "/home/user/デスクトップ/finetrainers/train.py", line 70, in main
trainer.run()
File "/home/user/デスクトップ/finetrainers/finetrainers/trainer/sft_trainer/trainer.py", line 97, in run
raise e
File "/home/user/デスクトップ/finetrainers/finetrainers/trainer/sft_trainer/trainer.py", line 92, in run
self._train()
File "/home/user/デスクトップ/finetrainers/finetrainers/trainer/sft_trainer/trainer.py", line 471, in _train
pred, target, sigmas = self.model_specification.forward(
File "/home/user/デスクトップ/finetrainers/finetrainers/models/wan/base_specification.py", line 316, in forward
pred = transformer(
File "/home/user/デスクトップ/finetrainers/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/user/デスクトップ/finetrainers/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1845, in _call_impl
return inner()
File "/home/user/デスクトップ/finetrainers/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1793, in inner
result = forward_call(*args, **kwargs)
File "/home/user/デスクトップ/finetrainers/.venv/lib/python3.10/site-packages/diffusers/models/transformers/transformer_wan.py", line 424, in forward
temb, timestep_proj, encoder_hidden_states, encoder_hidden_states_image = self.condition_embedder(
File "/home/user/デスクトップ/finetrainers/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/user/デスクトップ/finetrainers/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user/デスクトップ/finetrainers/finetrainers/patches/models/wan/patch.py", line 26, in _patched_WanTimeTextImageEmbedding_forward
temb = self.time_embedder(timestep).type_as(encoder_hidden_states)
File "/home/user/デスクトップ/finetrainers/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/user/デスクトップ/finetrainers/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user/デスクトップ/finetrainers/.venv/lib/python3.10/site-packages/diffusers/models/embeddings.py", line 1310, in forward
sample = self.linear_1(sample)
File "/home/user/デスクトップ/finetrainers/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/user/デスクトップ/finetrainers/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user/デスクトップ/finetrainers/.venv/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 125, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 must have the same dtype, but got BFloat16 and Float

Training steps: 0%| | 0/30000 [06:49<?, ?it/s]

• echo -ne '-------------------- Finished executing script --------------------\n\n'
  -------------------- Finished executing script --------------------

(.venv) (base) user@user-Super-Server:~/デスクトップ/finetrainers$

[ootsuka-repos]

train.sh
#!/bin/bash

set -e -x

export TORCH_LOGS="+dynamo,recompiles,graph_breaks"

export TORCHDYNAMO_VERBOSE=1

export WANDB_MODE="offline"
export NCCL_P2P_DISABLE=1
export TORCH_NCCL_ENABLE_MONITORING=0
export FINETRAINERS_LOG_LEVEL="DEBUG"

Finetrainers supports multiple backends for distributed training. Select your favourite and benchmark the differences!

BACKEND="accelerate"

BACKEND="ptd"

In this setting, I'm using 2 GPUs on a 4-GPU node for training

NUM_GPUS=2
CUDA_VISIBLE_DEVICES="0,1"

Check the JSON files for the expected JSON format

TRAINING_DATASET_CONFIG="training.json"
VALIDATION_DATASET_FILE="validation.json"

Depending on how many GPUs you have available, choose your degree of parallelism and technique!

DDP_1="--parallel_backend $BACKEND --pp_degree 1 --dp_degree 1 --dp_shards 1 --cp_degree 1 --tp_degree 1"
DDP_2="--parallel_backend $BACKEND --pp_degree 1 --dp_degree 2 --dp_shards 1 --cp_degree 1 --tp_degree 1"
DDP_4="--parallel_backend $BACKEND --pp_degree 1 --dp_degree 4 --dp_shards 1 --cp_degree 1 --tp_degree 1"
FSDP_2="--parallel_backend $BACKEND --pp_degree 1 --dp_degree 1 --dp_shards 2 --cp_degree 1 --tp_degree 1"
FSDP_4="--parallel_backend $BACKEND --pp_degree 1 --dp_degree 1 --dp_shards 4 --cp_degree 1 --tp_degree 1"
HSDP_2_2="--parallel_backend $BACKEND --pp_degree 1 --dp_degree 2 --dp_shards 2 --cp_degree 1 --tp_degree 1"

Parallel arguments

parallel_cmd=(
$DDP_2
)

Model arguments

model_cmd=(
--model_name "wan"
--pretrained_model_name_or_path "Wan-AI/Wan2.1-T2V-1.3B-Diffusers"
)

Dataset arguments

Here, we know that the dataset size if about ~50 videos. Since we're using 2 GPUs, we precompute

embeddings of 25 dataset items per GPU. Also, we're using a very small dataset for finetuning, so

we are okay with precomputing embeddings once and re-using them without having to worry about disk

space. Currently, however, every new training run performs precomputation even if it's not required

(which is something we've to improve [TODO(aryan)])

dataset_cmd=(
--dataset_config $TRAINING_DATASET_CONFIG
--dataset_shuffle_buffer_size 10
--enable_precomputation
--precomputation_items 20
--precomputation_once
)

Dataloader arguments

dataloader_cmd=(
--dataloader_num_workers 0
)

Diffusion arguments

diffusion_cmd=(
--flow_weighting_scheme "logit_normal"
)

Training arguments

We target just the attention projections layers for LoRA training here.

You can modify as you please and target any layer (regex is supported)

training_cmd=(
--training_type "lora"
--seed 42
--batch_size 1
--train_steps 30000
--rank 32
--lora_alpha 32
--target_modules "blocks.*(to_q|to_k|to_v|to_out.0)"
--gradient_accumulation_steps 1
--gradient_checkpointing
--checkpointing_steps 500
--checkpointing_limit 2

--resume_from_checkpoint 3000

--enable_slicing
--enable_tiling
)

Optimizer arguments

optimizer_cmd=(
--optimizer "adamw"
--lr 5e-5
--lr_scheduler "constant_with_warmup"
--lr_warmup_steps 1000
--lr_num_cycles 1
--beta1 0.9
--beta2 0.99
--weight_decay 1e-4
--epsilon 1e-8
--max_grad_norm 1.0
)

Validation arguments

validation_cmd=(
--validation_dataset_file "$VALIDATION_DATASET_FILE"
--validation_steps 500000
)

Miscellaneous arguments

miscellaneous_cmd=(
--tracker_name "finetrainers-wan"
--output_dir "wan"
--init_timeout 600
--nccl_timeout 600
--report_to "wandb"
)

Execute the training script

if [ "$BACKEND" == "accelerate" ]; then

ACCELERATE_CONFIG_FILE=""
if [ "$NUM_GPUS" == 1 ]; then
ACCELERATE_CONFIG_FILE="accelerate_configs/uncompiled_1.yaml"
elif [ "$NUM_GPUS" == 2 ]; then
ACCELERATE_CONFIG_FILE="accelerate_configs/uncompiled_2.yaml"
elif [ "$NUM_GPUS" == 4 ]; then
ACCELERATE_CONFIG_FILE="accelerate_configs/uncompiled_4.yaml"
elif [ "$NUM_GPUS" == 8 ]; then
ACCELERATE_CONFIG_FILE="accelerate_configs/uncompiled_8.yaml"
fi

accelerate launch --config_file "$ACCELERATE_CONFIG_FILE" --gpu_ids $CUDA_VISIBLE_DEVICES train.py
"${parallel_cmd[@]}"
"${model_cmd[@]}"
"${dataset_cmd[@]}"
"${dataloader_cmd[@]}"
"${diffusion_cmd[@]}"
"${training_cmd[@]}"
"${optimizer_cmd[@]}"
"${validation_cmd[@]}"
"${miscellaneous_cmd[@]}"

elif [ "$BACKEND" == "ptd" ]; then

export CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES

torchrun
--standalone
--nnodes=1
--nproc_per_node=$NUM_GPUS
--rdzv_backend c10d
--rdzv_endpoint="localhost:0"
train.py
"${parallel_cmd[@]}"
"${model_cmd[@]}"
"${dataset_cmd[@]}"
"${dataloader_cmd[@]}"
"${diffusion_cmd[@]}"
"${training_cmd[@]}"
"${optimizer_cmd[@]}"
"${validation_cmd[@]}"
"${miscellaneous_cmd[@]}"
fi

echo -ne "-------------------- Finished executing script --------------------\n\n"

[ootsuka-repos]

training.json
{
"datasets": [
{
"data_root": "/home/user/デスクトップ/finetrainers/nsfw_anime",
"dataset_type": "video",
"id_token": "nsfw_anime",
"video_resolution_buckets": [
[49, 512, 768],
[49, 768, 512]
],
"reshape_mode": "bicubic",
"remove_common_llm_caption_prefixes": true
}
]
}

[ootsuka-repos]

(.venv) (base) user@user-Super-Server:~/デスクトップ/finetrainers$ nvidia-smi
Fri Mar 14 18:55:33 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14 Driver Version: 550.54.14 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4090 D Off | 00000000:18:00.0 On | Off |
| 30% 38C P0 48W / 425W | 1041MiB / 49140MiB | 4% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 4090 D Off | 00000000:86:00.0 Off | Off |
| 30% 37C P0 53W / 425W | 11MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 2473 G /usr/lib/xorg/Xorg 424MiB |
| 0 N/A N/A 2689 G /usr/bin/gnome-shell 96MiB |
| 0 N/A N/A 94419 G ...yOnDemand --variations-seed-version 132MiB |
| 0 N/A N/A 104088 G ...seed-version=20250313-182214.581000 207MiB |
| 0 N/A N/A 108851 G ...erProcess --variations-seed-version 138MiB |
| 1 N/A N/A 2473 G /usr/lib/xorg/Xorg 4MiB |
+-----------------------------------------------------------------------------------------+
(.venv) (base) user@user-Super-Server:~/デスクトップ/finetrainers$

[ootsuka-repos]

Please let us know if we are missing any information you need.

[tolgacangoz]

I guess, now, this is another problem. So, could you please open another issue with the same info you just shared? Also, could you enclose each piece of information in three backticks: https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks

[a-r-r-o-w]

The initial error that you got regarding get_lr_scheduler_state() is because accelerate support is in a broken state at the moment. I'm working on fixing this before next release. For now, only torchrun will work.

Regarding the latest error you get with Wan, it is because of a bug introduced in #316. That PR fixed the behaviour for fp8 but broken the bf16 behaviour. I'll fix it asap.