Flux Outpainting

[fakerybakery]

Is your feature request related to a problem? Please describe.
Outpainting doesn't seem to be supported in Flux

Describe the solution you'd like.
Add outpainting to Flux

Describe alternatives you've considered.
N/A

Additional context.
#9135 introduces Flux inpainting

[asomoza]

Hi, we don't have any outpainting pipeline for any model, this is because outpainting can be done with the same inpainting pipeline you're referencing.

Outpainting is just inpainting but using just a part of the reference image to expand it.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[yiyixuxu]

closing this issue now
we may publish an outputting guide with flux in the future when we have more tools available & when @asomoza has good ideas!

[Clement-Lelievre]

Hi, we don't have any outpainting pipeline for any model, this is because outpainting can be done with the same inpainting pipeline you're referencing.

Outpainting is just inpainting but using just a part of the reference image to expand it.

(using diffusers @ 14a1b86)

hi @asomoza

maybe I'm doing this wrong but I'm not getting great results trying to do outpainting with the FluxInpaintPipeline (and either flux schnell or dev).

Here are the steps I follow:

1. take an input image
2. pad it (to try to help the model I use the patchmatch algo to pad)
3. build mask
4. blur mask a little to get a smoother transition / or use feathering for better blending
5. infer

Input image (1024*1024)

Input image + patch-match padding (1296*1360)

(Blurred) Mask (1296*1360)

Inference (prompt:"sky and stars", 28 steps, flux-dev, guidance 3, strength 0.8) (1296*1360)

In the result above, while there is some degree of prompt fidelity, the transition is awful.

What am I doing wrong?

thanks

[asomoza]

Hi, I don't think you're doing anything wrong but currently this is a limitation of the model, you can search for the issues with flux and inpainting.

This is basically because:

• Flux doesn't have an inpainting model.
• Flux is distilled and really aesthetic (so it's hard that it does smooth transitions).
• It has a really low range of the strength you can use and get some decent results (which is really bad for inpainting).

Also you don't get that much benefit with a blurred mask with the inpainting mask since the pipeline transforms it to a binary mask.

There are still some couple of things you can try:

• Use a undistilled model but you'll need to modify the pipeline so you can use CFG.
• Use the inpainting controlnet but it will probably change the base image too.
• Use differential diffusion with a soft/blurred mask, but the pipeline it's still a PR.

[Clement-Lelievre]

Hi, I don't think you're doing anything wrong but currently this is a limitation of the model, you can search for the issues with flux and inpainting.

This is basically because:

* Flux doesn't have an inpainting model.

* Flux is distilled and really aesthetic (so it's hard that it does smooth transitions).

* It has a really low range of the strength you can use and get some decent results (which is really bad for inpainting).

Also you don't get that much benefit with a blurred mask with the inpainting mask since the pipeline transforms it to a binary mask.

There are still some couple of things you can try:

* Use a undistilled model but you'll need to modify the pipeline so you can use CFG.

* Use the [inpainting controlnet](https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Beta) but it will probably change the base image too.

* Use differential diffusion with a soft/blurred mask, but the pipeline it's still a [PR](https://github.com/huggingface/diffusers/pull/9268).

@asomoza I tried the new community pipeline differential diffusion, thanks for pointing out it works quite well in my experience
(For large outpainting though, the model tends to just add a += uniform background)

[asomoza]

@Clement-Lelievre glad it worked better for you. Maybe for large outpainting you use other solutions and then do a second pass with Flux over it to finish the details.

For example a did a guide that fffiloni used as a base to do a cool space that does really good outpainting but since it was with SDXL and a lighting model, the final generation is not that good, you can use something like that and then do a img2img final pass over it.

Basically what I'm suggesting is that you use that technique with SDXL to pre fill the image with what you want before using Flux over it, that should prevent the uniform background.

[rubi-du]

@Clement-Lelievre Hi, I want to outpainting with flux too, but the result was very bad, the code as follow:
pipe = FluxDifferentialImg2ImgPipeline.from_pretrained( flux_dir, transformer=transformer, controlnet=controlnet, text_encoder_2=text_encoder_2, torch_dtype=torch.bfloat16 ) pipeline( prompt=prompt, image=image, mask_image=mask, strength=0.9, num_inference_steps=28, )

The image is:

The mask is:

But the result is:

What am I doing wrong?

thanks

[Clement-Lelievre]

@Clement-Lelievre Hi, I want to outpainting with flux too, but the result was very bad, the code as follow: pipe = FluxDifferentialImg2ImgPipeline.from_pretrained( flux_dir, transformer=transformer, controlnet=controlnet, text_encoder_2=text_encoder_2, torch_dtype=torch.bfloat16 ) pipeline( prompt=prompt, image=image, mask_image=mask, strength=0.9, num_inference_steps=28, )

The image is: The mask is: But the result is:

What am I doing wrong?

thanks

hey,
have a look at this app maybe this can inspire you

[asomoza]

now you can just use flux fill which was trained to do this, I tested it and it does a decent outpainting.

[rubi-du]

now you can just use flux fill which was trained to do this, I tested it and it does a decent outpainting.

Oh Thanks, so exciting, I found the api doc, but only has inpainting demo, Have any demos related to outpainting？

[asomoza]

I can give you one, here's the images and result:

source	mask	result

Here's the code:

import torch

from diffusers import FluxFillPipeline
from diffusers.utils import load_image


image = load_image("https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/flux_fill/peer_out.png")
mask = load_image("https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/flux_fill/peer_mask.png")

repo_id = "black-forest-labs/FLUX.1-Fill-dev"


pipe = FluxFillPipeline.from_pretrained(
    repo_id, torch_dtype=torch.bfloat16,
).to("cuda")

image = pipe(
    prompt="",
    image=image,
    mask_image=mask,
    guidance_scale=3.5,
    num_inference_steps=50,
    generator=torch.Generator("cpu").manual_seed(43),
).images[0]

image.save("outpaint.png")

Note: Until the next release, you'll need to install diffusers from main and also you need to provide the hf token to be able to use the model since it's gated.

[rubi-du]

guidance_scale=3.5,

@asomoza Woo, Thank you so much, it works all right