[fix] sync ControlNet-SD3 attention parameters of dual_attention_layer and qk_norm with SD3

[huchenz1]

When initializing ControlNet-SD3 with the from_transformer method, the dual_attention_layers and qk_norm parameters were not defined in the init method, resulting in the following error:

[rank1]:   File "train_controlnet_sd3.py", line 989, in main
[rank1]:     controlnet = SD3ControlNetModel.from_transformer(transformer)
[rank1]:   File "/home/23124047r/anaconda3/envs/py38/lib/python3.8/site-packages/diffusers/models/controlnet_sd3.py", line 251, in from_transformer
[rank1]:     controlnet = cls(**config)
[rank1]:   File "/home/23124047r/anaconda3/envs/py38/lib/python3.8/site-packages/diffusers/configuration_utils.py", line 665, in inner_init
[rank1]:     init(self, *args, **init_kwargs)
[rank1]: TypeError: __init__() got an unexpected keyword argument 'dual_attention_layers'

To resolve this, the missing keyword arguments were added to ensure compatibility with the from_transformer method.

And pos_embed_input for conditioning images needs to be the same as it in original transformers, for the images shape are the same.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yiyixuxu]

Is this change not related to SD 3.5?

[huchenz1]

It is releated to whether the control net is used for inpainting.
The control image for inpainiting has additional mask channel, but the normal controlnet does not need it. src/diffusers/pipelines/controlnet_sd3/pipeline_stable_diffusion_3_controlnet_inpainting.py

[yiyixuxu]

why are we removing this then? we can set it to 0 when it is not needed, no?

[huchenz1]

Sorry, you are right. I cancelled the commit and I changed the the default num_extra_conditioning_channels to 0 in the from_transformer method such that when using script, erros will not occur.

[yiyixuxu]

do we have to change the default? you can manually pass num_extra_conditioning_channels=0 when you need it to be 0, no?

otherwise I will tag the original author to see if it is ok to change here (to make sure it won't break anything)

[huchenz1]

Because the method in training script use the default parameters and the script is for add control with normal image instead of inpainting. As the result, with num_extra_conditioning_channels=0 in this method, script will occur errors.
In addtion, the default value of num_extra_conditioning_channels in ini method is also 0.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.