having a ton of issues since the recent updates

[dreammeai]

File "/content/diffusers/examples/dreambooth/train_dreambooth.py", line 852, in
main()
File "/content/diffusers/examples/dreambooth/train_dreambooth.py", line 726, in main
accelerator.clip_grad_norm_(params_to_clip, args.max_grad_norm)
File "/usr/local/lib/python3.8/dist-packages/accelerate/accelerator.py", line 920, in clip_grad_norm_
self.unscale_gradients()
File "/usr/local/lib/python3.8/dist-packages/accelerate/accelerator.py", line 904, in unscale_gradients
self.scaler.unscale_(opt)
File "/usr/local/lib/python3.8/dist-packages/torch/cuda/amp/grad_scaler.py", line 282, in unscale_
optimizer_state["found_inf_per_device"] = self.unscale_grads(optimizer, inv_scale, found_inf, False)
File "/usr/local/lib/python3.8/dist-packages/torch/cuda/amp/grad_scaler.py", line 210, in unscale_grads
raise ValueError("Attempting to unscale FP16 gradients.")
ValueError: Attempting to unscale FP16 gradients.
0% 0/1500 [00:02<?, ?it/s]
Traceback (most recent call last):
File "/usr/local/bin/accelerate", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/accelerate/commands/accelerate_cli.py", line 43, in main
args.func(args)
File "/usr/local/lib/python3.8/dist-packages/accelerate/commands/launch.py", line 837, in launch_command
simple_launcher(args)
File "/usr/local/lib/python3.8/dist-packages/accelerate/commands/launch.py", line 354, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '/content/diffusers/examples/dreambooth/train_dreambooth.py', '--stop_text_encoder_training=250', '--image_captions_filename', '--train_only_unet', '--save_starting_step=500', '--save_n_steps=1000', '--Session_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/fam4', '--pretrained_model_name_or_path=/content/stable-diffusion-custom', '--instance_data_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/fam4/instance_images', '--output_dir=/content/models/fam4', '--captions_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/fam4/captions', '--instance_prompt=', '--seed=668801', '--resolution=512', '--mixed_precision=fp16', '--train_batch_size=1', '--gradient_accumulation_steps=1', '--gradient_checkpointing', '--use_8bit_adam', '--learning_rate=1e-05', '--lr_scheduler=polynomial', '--lr_warmup_steps=0', '--max_train_steps=1500']' returned non-zero exit status 1.

[TheLastBen]

is that the default model ? or a custom one ?

[dreammeai]

i tried the default model and custom one. both gave them same error

[TheLastBen]

you're probably using a premium gpu, that issue was fixed in commit pushed yesterday

[dreammeai]

I tried it today and it gives a another error.

Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/diffusers/configuration_utils.py", line 372, in load_config
config_dict = cls._dict_from_json_file(config_file)
File "/usr/local/lib/python3.8/dist-packages/diffusers/configuration_utils.py", line 466, in _dict_from_json_file
return json.loads(text)
File "/usr/lib/python3.8/json/init.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.8/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 3 (char 284)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/content/diffusers/examples/dreambooth/train_dreambooth.py", line 852, in
main()
File "/content/diffusers/examples/dreambooth/train_dreambooth.py", line 563, in main
noise_scheduler = DDPMScheduler.from_config(args.pretrained_model_name_or_path, subfolder="scheduler")
File "/usr/local/lib/python3.8/dist-packages/diffusers/configuration_utils.py", line 193, in from_config
config, kwargs = cls.load_config(pretrained_model_name_or_path=config, return_unused_kwargs=True, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/diffusers/configuration_utils.py", line 374, in load_config
raise EnvironmentError(f"It looks like the config file at '{config_file}' is not a valid JSON file.")
OSError: It looks like the config file at '/content/stable-diffusion-custom/scheduler/scheduler_config.json' is not a valid JSON file.
Traceback (most recent call last):
File "/usr/local/bin/accelerate", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/accelerate/commands/accelerate_cli.py", line 43, in main
args.func(args)
File "/usr/local/lib/python3.8/dist-packages/accelerate/commands/launch.py", line 837, in launch_command
simple_launcher(args)
File "/usr/local/lib/python3.8/dist-packages/accelerate/commands/launch.py", line 354, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '/content/diffusers/examples/dreambooth/train_dreambooth.py', '--stop_text_encoder_training=250', '--image_captions_filename', '--train_only_unet', '--save_starting_step=500', '--save_n_steps=0', '--Session_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/fam4', '--pretrained_model_name_or_path=/content/stable-diffusion-custom', '--instance_data_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/fam4/instance_images', '--output_dir=/content/models/fam4', '--captions_dir=/content/gdrive/MyDrive/Fast-Dreambooth/Sessions/fam4/captions', '--instance_prompt=', '--seed=632025', '--resolution=512', '--mixed_precision=fp16', '--train_batch_size=1', '--gradient_accumulation_steps=1', '--gradient_checkpointing', '--use_8bit_adam', '--learning_rate=1e-05', '--lr_scheduler=polynomial', '--lr_warmup_steps=0', '--max_train_steps=1500']' returned non-zero exit status 1.
Something went wrong

[TheLastBen]

hopefully this will fix it : 3884ddb