SDXL InPainting: Mask blur option is negated by forced binarization.

[zacheryvaughn]

The SDXL InPainting pipeline's documentation suggests using pipeline.mask_processor.blur() for creating soft masks, but this functionality is effectively broken due to the implementation order. Please let me know if I'm missing something here. Based on my testing, whether I use a blurred mask or blur them with the built in method, they still show a solid seam as if there was no blur applied.

The mask processor is initialized with forced binarization:

self.mask_processor = VaeImageProcessor(
    vae_scale_factor=self.vae_scale_factor,
    do_normalize=False,
    do_binarize=True,  # Forces binarization
    do_convert_grayscale=True
)

When processing masks, any blur effect is applied before binarization, which then converts all values back to pure black and white, completely negating the blur effect (if I'm not mistaken). The VaeImageProcessor defaults binarize to false, but when masks are initialized it sets binarize to true.

Relevant files:
diffusers/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py[326]
diffusers/image_processor.py[88][276][523]

Reproduction

Inpaint pipeline config using either your own blurred image or the built in blur method according to documentation. It's fairly self explanatory and I don't have a minimal script to share.

System Info

diffusers==0.32.2

Who can help?

@yiyixuxu @DN6

[zacheryvaughn]

The binarization should happen before the blur effect is applied, right?
Or another option would be to disable binarization if the built in blur option is used.
Additionally, for people who already have binarization and blur effects built into the GUI, we should be able to set binarize to false from our end without having to modify the library.

At the end of the day, this is free open source software so I have no expectations. Thanks a lot, you guys are awesome.

[asomoza]

Hi, there's a confusion that probably we will need to fix in the docs if it's not clear.

The blur for the image mask is used when the inpainted part is pasted back in the original image, so going step by step:

First the mask is converted with the image proccesor into another variable called mask so the original mask_image it's still the same one:

diffusers/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py

Lines 1449 to 1451 in aad69ac

	mask = self.mask_processor.preprocess(
	mask_image, height=height, width=width, resize_mode=resize_mode, crops_coords=crops_coords
	)

After that, the inpainting pipeline uses the mask and not the image_mask, just to be clear, the inpainting models need a binary mask, they don't work with normal images, so the final tensor has to be 1s and 0s.

Finally, at the end, if the user passed a padding_mask_crop parameter, the pipeline uses the original mask to apply the overlay:

diffusers/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py

Lines 1731 to 1732 in aad69ac

	if padding_mask_crop is not None:
	image = [self.image_processor.apply_overlay(mask_image, original_image, i, crops_coords) for i in image]

Here's the apply_overlay function if you want to read it.

Additionally, for people who already have binarization and blur effects built into the GUI, we should be able to set binarize to false from our end without having to modify the library.

Since the blur can be done by the user and we always need a binary mask for the inpainting we do the binarize always to prevent errors.

We're working on a modular diffusers system where you will be able to do what you're asking though, so if you wait a little longer and insist on doing this on your side, you will be able to build a custom pipeline with whatever you want.

In the meantime, if you want to use gradient masks (soft inpainting) you can use differential-diffusion that actually works with what you want.

At the end of the day, this is free open source software so I have no expectations. Thanks a lot, you guys are awesome.

Even if you don't have expectations, please don't hesitate to open issues if something doesn't work or if you see errors in the code, we will fix it if it's something we missed and we try to always answer the questions.

[zacheryvaughn]

Big thanks for pointing me to diff diff, really awesome and exactly what I was looking for. I've modified it to add padding/scaling options similar to what the inPainting pipeline offers, and also created an outpainting variant.