Support ControlNet for Qwen-Image

[haofanwang]

What does this PR do?

Add ControlNet-Union (InstantX/Qwen-Image-ControlNet-Union) support for Qwen-Image.

Inference

import torch
from diffusers.utils import load_image
from diffusers import QwenImageControlNetPipeline, QwenImageControlNetModel

base_model = "Qwen/Qwen-Image"
controlnet_model = "InstantX/Qwen-Image-ControlNet-Union"

controlnet = QwenImageControlNetModel.from_pretrained(controlnet_model, torch_dtype=torch.bfloat16)

pipe = QwenImageControlNetPipeline.from_pretrained(
    base_model, controlnet=controlnet, torch_dtype=torch.bfloat16
)
pipe.to("cuda")

# canny
control_image = load_image("https://huggingface.co/InstantX/Qwen-Image-ControlNet-Union/resolve/main/conds/canny.png")
prompt = "Aesthetics art, traditional asian pagoda, elaborate golden accents, sky blue and white color palette, swirling cloud pattern, digital illustration, east asian architecture, ornamental rooftop, intricate detailing on building, cultural representation."
controlnet_conditioning_scale = 1.0

image = pipe(
    prompt=prompt,
    negative_prompt=" ",
    control_image=control_image,
    controlnet_conditioning_scale=controlnet_conditioning_scale,
    width=control_image.size[0],
    height=control_image.size[1],
    num_inference_steps=30,
    true_cfg_scale=4.0,
    generator=torch.Generator(device="cuda").manual_seed(42),
).images[0]
image.save(f"qwenimage_cn_union_result.png")

Multi-Conditions

import torch
from diffusers.utils import load_image
from diffusers import QwenImageControlNetPipeline, QwenImageControlNetModel, QwenImageMultiControlNetModel

base_model = "Qwen/Qwen-Image"
controlnet_model = "InstantX/Qwen-Image-ControlNet-Union"

controlnet = QwenImageControlNetModel.from_pretrained(controlnet_model, torch_dtype=torch.bfloat16)
controlnet = QwenImageMultiControlNetModel([controlnet])

pipe = QwenImageControlNetPipeline.from_pretrained(
    base_model, controlnet=controlnet, torch_dtype=torch.bfloat16
)
pipe.to("cuda")

# canny
control_image = load_image("https://huggingface.co/InstantX/Qwen-Image-ControlNet-Union/resolve/main/conds/canny.png")
prompt = "Aesthetics art, traditional asian pagoda, elaborate golden accents, sky blue and white color palette, swirling cloud pattern, digital illustration, east asian architecture, ornamental rooftop, intricate detailing on building, cultural representation."
controlnet_conditioning_scale = 1.0

# Please note that the results will not be identical, because the generator is called in different order.
image = pipe(
    prompt=prompt,
    negative_prompt=" ",
    control_image=[control_image, control_image],
    controlnet_conditioning_scale=[controlnet_conditioning_scale/2, controlnet_conditioning_scale/2],
    width=control_image.size[0],
    height=control_image.size[1],
    num_inference_steps=30,
    true_cfg_scale=4.0,
    generator=torch.Generator(device="cuda").manual_seed(42),
).images[0]
image.save(f"qwenimage_cn_union_multi_result_.png")

Sanity Check

[yiyixuxu]

ohh, do we not support multi-controlnets use case?

[yiyixuxu]

we can probably do something like this, and this way we don't need to something special for one union controlnet

Suggested change

	if len(self.nets) == 1:
	controlnet = self.nets[0]
	for i, (image, scale) in enumerate(zip(controlnet_cond, conditioning_scale)):
	block_samples = controlnet(
	if len(self.nets) == 1:
	controlnets = [self.nets] * len(controlnet_cond)
	for i, (image, scale, controlnet) in enumerate(zip(controlnet_cond, conditioning_scale, controlnets)):
	block_samples = controlnet(

[haofanwang]

@yiyixuxu Added support for multiple conditions. ruff check has passed.

[vuongminh1907]

Excellent work @haofanwang ! did you try controlnet with Flux Kontext too?

[yiyixuxu]

@bot /style

[github-actions]

Style bot fixed some files and pushed the changes.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yiyixuxu]

thanks!

[vladmandic]

page https://huggingface.co/InstantX/Qwen-Image-ControlNet-Union returns 404 ?!

[haofanwang]

Will be released soon.