FluxFillPipeline quality deteriorates over multiple iterations of inpainting

[svenschultze]

Describe the bug

When the FluxFillPipeline is run multiple times on the same image, the image quality deteriorates in each iteration.

Reproduction

import torch
from diffusers import FluxFillPipeline
from diffusers.utils import load_image

pipe = FluxFillPipeline.from_pretrained("black-forest-labs/FLUX.1-Fill-dev", torch_dtype=torch.bfloat16)
pipe = pipe.to("cuda")

img = load_image("https://huggingface.co/datasets/diffusers/diffusers-images-docs/resolve/main/cup.png")
mask = load_image("https://huggingface.co/datasets/diffusers/diffusers-images-docs/resolve/main/cup_mask.png")


def fill(prompt, image, mask):
    return pipe(
        prompt=prompt,
        image=image,
        mask_image=mask,
        guidance_scale=30,
        num_inference_steps=50,
        max_sequence_length=512,
        height=1632,
        width=1232,
        generator=torch.Generator("cpu").manual_seed(0),
    ).images[0]

prompt = "a white paper cup"
for i in range(12):
    print(i)
    img = fill(prompt, img, mask)
    img.save(f"output_{i}.png")

Logs

System Info

Diffusers Version 0.31.0
Python Version 3.10
CUDA Version 12.1.0
DGX H100

Who can help?

@yiyixuxu @DN6 @aso

[asomoza]

the vae decoding/encoding is not a lossless process, also the quality of flux fill is not the best, I found it likes to blur the fill part, so after each iteration you will get a less quality blurred image.

What you're seeing it's kind of expected and probably you will get the same results everywhere with the vanilla fill, you should probably paste back the inpainted part only over the original image to mitigate this issue.

[CyberVy]

You can use a different seed in every iteration. I think the problem will probably be solved.

AFAIK not only fill pipeline but also other image input pipelines have this issue.

Iteration with the same seed in the image input pipelines will 100% trigger off this issue.

I don't know if it is a bug or feature.
@DN6 @yiyixuxu

[svenschultze]

Thanks for the responses!

@asomoza Pasting the image works. Maybe this should be a parameter in the pipeline to simplify things?

@CyberVy I tried using random seeds each time, but this did not fix the issue. I am getting the same artifacts.

[CyberVy]

Thanks for the responses!

@asomoza Pasting the image works. Maybe this should be a parameter in the pipeline to simplify things?

@CyberVy I tried using random seeds each time, but this did not fix the issue. I am getting the same artifacts.

I just noticed your guidance scale, which always leads to artifacts when it is too high.
You can pass in a lower guidance scale, and have a try.

But even so, you may still get the artifacts, because 12 iterations are too many. And artifacts appearing in any of the 12 iterations will cause issues.

You can use different parameters and record the results of each iteration. By comparing these results, you can identify exactly which iteration caused the artifacts.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[hardikdava]

any update on this issue?

[asomoza]

what do you mean with update? As I wrote before, the vae encoding/decoding is not a lossless process, there's nothing we can do about this and it will happen in diffusers and in any other library or UI you use, it doesn't matter if you use a different seed, after a couple if iterations the image will always look bad.

[sucessfullys]

what do you mean with update? As I wrote before, the vae encoding/decoding is not a lossless process, there's nothing we can do about this and it will happen in diffusers and in any other library or UI you use, it doesn't matter if you use a different seed, after a couple if iterations the image will always look bad.

Image loos bad, how do you slove? I meet this problem.

[hardikdava]

@sucessfullys the problem is as @asomoza mentioned about VAE is not lossless process. After each iteration it in theory degrade 10% of image quality.