Skip to content

Validation does not work with Anima LoRa #2286

@Hirmuolio

Description

@Hirmuolio

Trying to have validation loss while training Anima LoRa does not work.

Enabling validation loss with validation_split = 0.5 and validate_every_n_steps = 4 results in the following error:

traceback (most recent call last):
  File "/ai/sd-scripts/anima_train_network.py", line 448, in <module>
    trainer.train(args)
  File "/ai/sd-scripts/train_network.py", line 1558, in train
    loss = self.process_batch(
           ^^^^^^^^^^^^^^^^^^^
  File "/ai/sd-scripts/anima_train_network.py", line 353, in process_batch
    text_encoder_outputs_list = anima_text_encoding_strategy.drop_cached_text_encoder_outputs(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: AnimaTextEncodingStrategy.drop_cached_text_encoder_outputs() missing 1 required positional argument: 't5_attn_mask'
steps:   0%|                                                                                     | 4/5000 [00:12<4:18:41,  3.11s/it, avr_loss=0.0595]
validation steps:   0%|▏                                                           | 1/300 [00:01<08:13,  1.65s/it, val_avg_loss=0.102, timestep=200]
Traceback (most recent call last):
  File "/ai/kohya_ss/.venv/bin/accelerate", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/ai/kohya_ss/.venv/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 50, in main
    args.func(args)
  File "/ai/kohya_ss/.venv/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1199, in launch_command
    simple_launcher(args)
  File "/ai/kohya_ss/.venv/lib/python3.11/site-packages/accelerate/commands/launch.py", line 785, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/ai/kohya_ss/.venv/bin/python', 'anima_train_network.py', '--config_file=/ai/training/anima_test.toml']' returned non-zero exit status 1.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions