Question about control sd3

[Henry-Bi]

I've noticed a potential inconsistency in how the VAE-encoded control_image is processed between the training script for ControlNet with Stable Diffusion 3 and the corresponding inference pipeline.

In the inference pipeline (pipeline_stable_diffusion_3_controlnet.py):

The control_image latent is processed by both subtracting the vae_shift_factor and multiplying by the scaling_factor.

diffusers/src/diffusers/pipelines/controlnet_sd3/pipeline_stable_diffusion_3_controlnet.py

Line 1074 in 425a715

control_image = (control_image - vae_shift_factor) * self.vae.config.scaling_factor

In the training script (train_controlnet_sd3.py):

However, in the provided training example, the VAE-encoded controlnet_image is only multiplied by the scaling_factor, without subtracting the shift_factor.

diffusers/examples/controlnet/train_controlnet_sd3.py

Line 1333 in 425a715

controlnet_image = controlnet_image * vae.config.scaling_factor

Shouldn't the training script also apply the vae_shift_factor to maintain consistency with the inference process? It seems the correct implementation in the training script should be: controlnet_image = (controlnet_image - vae.config.shift_factor) * vae.config.scaling_factor

[viv92]

Same question.

[asomoza]

cc: @sayakpaul

[sayakpaul]

Yeah it should be scaled according to how the inference pipelines are set up. Would you like to open a PR?

[Henry-Bi]

Yeah it should be scaled according to how the inference pipelines are set up. Would you like to open a PR?

Happy to help. I've gone ahead and opened the PR. Looking forward to your feedback!

[asomoza]

closing because it was resolved with #11909