Z_Image Fun ControlNet V2 Inpaint Mode: need examples please

[deepbeepmeep]

Hi

I have implemented Z_Image Fun ControlNet V2 in WanGP and optimized to use much less VRAM. Control Net seems to work properly as I can transfer a pose for instance. However I have some weird results with the Inpaint mode.

It is the Inpaint mode full working ? I would be grateful if you could give the semantics of the 33 latent input channels when using inpainting. For instance what should contain the masked aread (white, black, original image?) in the Inpainted Image It would be great to have examples.

[bubbliiiing]

Hi, are you familiar with this section of the code?

https://github.com/aigc-apps/VideoX-Fun/blob/main/examples/z_image_fun/predict_i2i_inpaint_2.0.py

[deepbeepmeep]

@bubbliiiing I had a look at indeed. But unless I am mistaken this part of the code doesnt explain if there is any code color to be used in the inpainting image. For instance with Vace, grey 127 was used to designate the area to be inpainted (even if an additonal mask was provided). Also, your home page seems to imply that reference to image is supported. Right now I have been unable to do that whether the reference person is in the control image or in the inpainting image.

Sadly as I have been unable to get the inpainting to work properly I had to disable in WanGP.

It would help if you could provide a couple of examples of control, inpainting and mask fed to predict_i2i_inpaint_2.0.py with the expected output image.

[bubbliiiing]

There are 33 channels in total. The first 16 channels are for control, the 17th channel is the mask, and the last 16 channels are the masked image. In the pipeline, the masked regions are set to 0 in latent space, which corresponds to a value of 127 in the integer image domain.

[bubbliiiing]

I'm not sure if this is the information you're looking for. As for the inpainting capability, it's roughly like this - it can be input together with the control. This is the result of the latest 8steps model, not the old 2.0, but the old 2.0 has similar functionality as well.

[deepbeepmeep]

Thank you for these explanations.

In your example, the inpaint / masked image (top left ?) has no pixel set grey 127 in what should be the masked area. Is the conversion to grey supposed to happen later in the preprocessing pipeline.

Assuming the girl in the inpaint image is replaced with gray pixels how come she ends up anyway in the output image ? Is she described in the prompt ?