[Fix] remove setting lr for T5 text encoder when using prodigy in flux dreambooth lora script (huggingface#9473)

* fix: removed setting of text encoder lr for T5 as it's not being tuned

* fix: removed setting of text encoder lr for T5 as it's not being tuned

---------

Co-authored-by: Sayak Paul <[email protected]>
Co-authored-by: Linoy Tsaban <[email protected]>

Author: 3 people

examples/dreambooth/train_dreambooth_flux.py

@@ -1288,7 +1288,6 @@ def load_model_hook(models, input_dir):
             # changes the learning rate of text_encoder_parameters_one and text_encoder_parameters_two to be
             # --learning_rate
             params_to_optimize[1]["lr"] = args.learning_rate
-            params_to_optimize[2]["lr"] = args.learning_rate
 
         optimizer = optimizer_class(
             params_to_optimize,

examples/dreambooth/train_dreambooth_lora_flux.py

@@ -1370,7 +1370,6 @@ def load_model_hook(models, input_dir):
             # changes the learning rate of text_encoder_parameters_one and text_encoder_parameters_two to be
             # --learning_rate
             params_to_optimize[1]["lr"] = args.learning_rate
-            params_to_optimize[2]["lr"] = args.learning_rate
 
         optimizer = optimizer_class(
             params_to_optimize,