Fix Flux Controlnet Pipeline _callback_tensor_inputs Missing Some Elements

[CyberVy]

._callback_tensor_inputs of Flux controlnet related pipelines are missing some important elements.

This PR is to fix it.

I'll be very appreciated if you can have a check.
@asomoza

[CyberVy]

I've also accidentally noticed that Flux related pipeline classes are missing some base classes.

class FluxPipeline(
    DiffusionPipeline,
    FluxLoraLoaderMixin,
    FromSingleFileMixin,
    TextualInversionLoaderMixin,
    FluxIPAdapterMixin,
):...
class FluxImg2ImgPipeline(DiffusionPipeline, FluxLoraLoaderMixin, FromSingleFileMixin, FluxIPAdapterMixin):...
class FluxInpaintPipeline(DiffusionPipeline, FluxLoraLoaderMixin, FluxIPAdapterMixin):...

I'll fix it in another PR.

[asomoza]

oh that's nice, never tried changing the mask of the inpainting on a callback, could lead to some interesting results. Did you tried doing that?

Checking the XL one, they do have the mask inputs but nevertheless, I've never tried doing something with it.

Also, the other inpaint pipelines have the masked_image_latents is there a reason you didn't add it here?

[CyberVy]

@asomoza

oh that's nice, never tried changing the mask of the inpainting on a callback, could lead to some interesting results. Did you tried doing that?

Yes, it makes it possible to make a CFG cutoff callback function for inpainting pipeline.

def f(pipeline, i, t, callback_kwargs):
    cutoff_step = int(pipeline.num_timesteps * 0.5)
    if i == cutoff_step:
        prompt_embeds = callback_kwargs["prompt_embeds"]
        prompt_embeds = prompt_embeds[-1:]

        add_text_embeds = callback_kwargs["add_text_embeds"]
        add_text_embeds = add_text_embeds[-1:]

        add_time_ids = callback_kwargs["add_time_ids"]
        add_time_ids = add_time_ids[-1:]

        mask = callback_kwargs["mask"]
        mask = mask[-1:]
        pipeline._guidance_scale = 0.0
        callback_kwargs["prompt_embeds"] = prompt_embeds
        callback_kwargs["add_text_embeds"] = add_text_embeds
        callback_kwargs["add_time_ids"] = add_time_ids
        callback_kwargs["mask"] = mask
    return callback_kwargs

Also, the other inpaint pipelines have the masked_image_latents is there a reason you didn't add it here?

Because it does nothing in Flux inpainting denoising loop, which means any change to masked_image_latents is meaningless.
However you can retrieve and observe it from a callback function. And considering the uniform API standard, I think it's fine adding masked_image_latents to _callback_tensor_inputs. What is your opinion?

[asomoza]

yeah, lets keep it uniform with the rest of the API, even if it's not used right now.

[CyberVy]

yeah, lets keep it uniform with the rest of the API, even if it's not used right now.

👌
And I'll open another PR to add mask and masked_image_latents for StableDiffusionControlNetInpaintPipeline. I didn't noticed it in last fix. 😭

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[asomoza]

thanks! Failing test are not related to this PR.