A compatibility issue when using custom Stable Diffusion with pre-trained ControlNets

[ScienceLi1125]

I have successfully fine-tuned a Stable Diffusion v1.5 model using the Dreambooth script, and the results are excellent. However, I've encountered a compatibility issue when using this custom model with pre-trained ControlNets. Since the Dreambooth process modifies the U-Net weights, the original ControlNet is no longer aligned with the fine-tuned model, leading to a significant degradation in control and image quality.

My goal is to find a way to make them compatible again. It's important to clarify that I am trying to avoid a full, separate fine-tuning of the ControlNet on my custom model. That process is data- and resource-intensive, which defeats the purpose of a lightweight personalization method like Dreambooth. I have tried modifying the train_dreambooth.py script to incorporate ControlNet, but results have been consistently poor.

Is there a dedicated script or a recommended workflow in diffusers to fine-tune a Stable Diffusion with ControlNet via Dreambooth? Any guidance or pointers would be greatly appreciated. Thanks a lot!

[asomoza]

Hi, I have never encountered a DB model or lora that made the controlnets not work, the only cases where I know that controlnet stopped working was when people fully finetuned with a large amount of data the base models, for example, from SDXL to Pony, or Illustrous.

I don't know what you did, but if the controlnet stopped working you probably really overfitted the model in which case you should retrain with more sensible parameters. Dreambooth is a really quick and simple finetuning that shouldn't produce any controlnet degradation.

[ScienceLi1125]

Hi, I have never encountered a DB model or lora that made the controlnets not work, the only cases where I know that controlnet stopped working was when people fully finetuned with a large amount of data the base models, for example, from SDXL to Pony, or Illustrous.

I don't know what you did, but if the controlnet stopped working you probably really overfitted the model in which case you should retrain with more sensible parameters. Dreambooth is a really quick and simple finetuning that shouldn't produce any controlnet degradation.

Thank you for your instant reply! I apologize if my initial description was ambiguous. To clarify, the issue is not that ControlNet completely stops working or crashes. It still runs, but there is a noticeable degradation in the quality of control. The fine-tuned model does not adhere to the ControlNet's conditions as precisely as the original SDv1.5 model does.

For instance, with a Canny ControlNet, the generated content might bleed or extend beyond the boundaries defined by the Canny edges. Or with an OpenPose ControlNet, the final character's pose might not strictly conform to the input skeleton, deviating slightly from the intended posture.

This prevents the ControlNet from guiding the diffusion process with the high precision it has on the original U-Net.

[asomoza]

yeah, when I wrote stopped working, I meant that the controlnet didn't produce any good results in the generation, meaning something similar to what you're experiencing.

I still say the same, the only time that I know this happens is when people fine tune the base model with millions of images in a completely different style, meaning that, the model is mostly like trained from scratch. This is not something that should happen with a simple dreambooth training.

You also are not providing that much information, but I repeat again, there shouldn't be any controlnet degradation with a simple dreambooth training, probably you overfitted the model and maybe you're training on something that overrides and goes against the controlnet.

Also canny is the weakest of the controlnets but its been a while since I used seriously SD 1.5, don't really know if there are any good controlnets that compare to the modern ones like ControlnetUnion.

I'll answer your original question then, no, there isn't a script or anything remotely similar to train dreambooth with controlnet because they are completely different training objectives, there isn't even a training script to finetune the model with controlnet, you either finetune the model or train a controlnet, never both at the same time.

[ScienceLi1125]

Thank you for your reply. Your answer solved my doubt. Since it is impossible to finetune SD and ControlNet at the same time, I will give up this approach. Thank you again for your timely reply!

[asomoza]

no problem, hope you can find a good method to achieve what you're trying to do.

[asomoza]

I'll close this issue but if you have any more questions feel free to ask but just so you know, we have a discussions tab with a specific Q&A category.