FluxImg2ImgPipeline produces output almost identical to the input image

[CyberVy]

Describe the bug

When providing an input image and a prompt, the resulting output image is nearly identical to the input image, regardless of how I adjust the parameters. The differences are so minor that they are almost imperceptible.

Even after setting the strength parameter to a very high value, the output still doesn't seem to reflect the given prompt. It appears as though the prompt is not being considered in the generation process.

Reproduction

from diffusers import FluxPipeline,FluxImg2ImgPipeline
import torch


# the transformer model is from https://huggingface.co/Kijai/flux-fp8/blob/main/flux1-dev-fp8.safetensors
pipeline = FluxPipeline(...)
pipeline.enable_model_cpu_offload()

prompt = "a cat with wing flying in the sky"
num_inference_steps = 25
guidance_scale = 3
seed = 15827082008518028345
image = pipeline(prompt=prompt,num_inference_steps=num_inference_steps,guidance_scale=guidance_scale,
                 generator=torch.Generator(pipeline.device).manual_seed(seed)).images[0]


i2i_pipeline = FluxImg2ImgPipeline(**pipeline.components)
i2i_pipeline.enable_model_cpu_offload()

i2i_prompt = "in a forest"
i2i_seed = 888
strength = 0.7
i2i_image = i2i_pipeline(image=image,prompt=i2i_prompt,strength=strength,num_inference_steps=num_inference_steps,guidance_scale=guidance_scale,
                         generator=torch.Generator(i2i_pipeline.device).manual_seed(i2i_seed)).images[0]

image:

i2i_image:

As you can see, the result from image to image pipeline is quite simliar to the input image, however, the strength is 0.7.

Logs

No response

System Info

• 🤗 Diffusers version: 0.32.0.dev0
• Platform: Linux-6.6.56+-x86_64-with-glibc2.35
• Running on Google Colab?: No
• Python version: 3.10.14
• PyTorch version (GPU?): 2.4.0 (True)
• Flax version (CPU?/GPU?/TPU?): 0.8.4 (gpu)
• Jax version: 0.4.26
• JaxLib version: 0.4.26.dev20240620
• Huggingface_hub version: 0.26.2
• Transformers version: 4.46.3
• Accelerate version: 1.1.1
• PEFT version: 0.14.0
• Bitsandbytes version: not installed
• Safetensors version: 0.4.5
• xFormers version: not installed
• Accelerator: Tesla T4, 15360 MiB
  Tesla T4, 15360 MiB
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

Who can help?

@yiyixuxu @DN6

[SoftologyPro]

I can confirm this issue, no matter what steps, strength or guidance scale settings I try the resulting image is always very close to the init image.

[CyberVy]

@sayakpaul Somebody came across the same issue. Could you please take a look? Thank you!

[CyberVy]

I can confirm this issue, no matter what steps, strength or guidance scale settings I try the resulting image is always very close to the init image.

@SoftologyPro Thanks for your response. It's been a month since I posted this issue, but no one replied. I almost thought no one have the same problem.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[CyberVy]

@SoftologyPro Are you still bothered by this bug? It seems it has been fixed in the latest version.

[SoftologyPro]

@SoftologyPro Are you still bothered by this bug? It seems it has been fixed in the latest version.

Still the same behavior here. Using the latest diffusers. Is that what you mean by "latest version"?
Can you share a script that img2img works for you so I can see what may be different with mine?

My code specific to img2img is

    pipe = FluxImg2ImgPipeline(
        scheduler=scheduler,
        text_encoder=text_encoder,
        tokenizer=tokenizer,
        text_encoder_2=text_encoder_2,
        tokenizer_2=tokenizer_2,
        vae=vae,
        transformer=transformer,
    ).to("cuda")

    generator = torch.Generator(pipe.device).manual_seed(args2.seed)
    image = pipe(
        image=load_image(args2.init_image),
        strength=args2.init_image_strength,
        prompt=args2.prompt, 
        width=args2.W,
        height=args2.H,
        num_inference_steps=args2.steps, 
        generator=generator,
        guidance_scale=3.5,
    ).images[0]

[CyberVy]

@SoftologyPro
My code is similar with yours. But I don't know the model you used.
What is the value of your strength parameter?
Sometimes, a high strength (0.8 - 1) is required, especially if your input image is quite different from your prompt.

My model is here.

num_inference_steps
20

guidance_scale
3.5

prompt
a cat in a forest

This is the input image.

With strength = 0.8

With strength = 0.9

With strength = 1.0

The results are acceptable.

[asomoza]

I remember that when I tested Flux quite a with img2img the range of the denoising strength was very limited, I don't remember the exact numbers now, by the values where in the range of the high 0.7s and low 0.9.

This is not an issue and AFAIK we didn't do something to fix it in the recent versions, this is something from the model and it's probably because it's a distilled version of the model, maybe you can try with the new Flux finetuned model that can use CFG and test this.

[SoftologyPro]

I remember that when I tested Flux quite a with img2img the range of the denoising strength was very limited, I don't remember the exact numbers now, by the values where in the range of the high 0.7s and low 0.9.

This is not an issue and AFAIK we didn't do something to fix it in the recent versions, this is something from the model and it's probably because it's a distilled version of the model, maybe you can try with the new Flux finetuned model that can use CFG and test this.

Can you give me a link to the new finetuned model and an example script?

[CyberVy]

Yes, I think maybe the distilled model is the reason.