[Feature Request] Add optional latent-input to reference_only for img2img

[MoonMoon82]

Hi! Could you please add an optional latent input for img2img process using the reference_only node?
This node is already awesome! Great work!

Kind regards

[addddd2]

I made my own as below

--- reference_only.py   2023-07-26 22:24:24.000000000 +0300
+++ reference_only2.py  2023-08-22 19:42:10.985318300 +0300
@@ -1,10 +1,11 @@
 import torch

-class ReferenceOnlySimple:
+class ReferenceOnlySimple2:
     @classmethod
     def INPUT_TYPES(s):
         return {"required": { "model": ("MODEL",),
                               "reference": ("LATENT",),
+                              "input2": ("LATENT",),
                               "batch_size": ("INT", {"default": 1, "min": 1, "max": 64})
                               }}

@@ -13,12 +14,12 @@

     CATEGORY = "custom_node_experiments"

-    def reference_only(self, model, reference, batch_size):
+    def reference_only(self, model, reference, input2, batch_size):
         model_reference = model.clone()
         size_latent = list(reference["samples"].shape)
         size_latent[0] = batch_size
-        latent = {}
-        latent["samples"] = torch.zeros(size_latent)
+        latent = input2
+        #latent["samples"] = torch.zeros(size_latent)

         batch = latent["samples"].shape[0] + reference["samples"].shape[0]
         def reference_apply(q, k, v, extra_options):
@@ -50,5 +51,5 @@
         return (model_reference, {"samples": out_latent, "noise_mask": torch.cat((out_mask, mask))})

 NODE_CLASS_MAPPINGS = {
-    "ReferenceOnlySimple": ReferenceOnlySimple,
+    "ReferenceOnlySimple2": ReferenceOnlySimple2,
 }

[MoonMoon82]

@addddd2
this pretty looks like my experiment ;-)
But as already said - The output looks more like the input image instead of the reference image.

import torch

class Inp_ReferenceOnlySimple:
    @classmethod
    def INPUT_TYPES(s):
        return {"required": { "model": ("MODEL",),
                              "reference": ("LATENT",),
                              "latent": ("LATENT",),
                              "batch_size": ("INT", {"default": 1, "min": 1, "max": 64})
                              }}

    RETURN_TYPES = ("MODEL", "LATENT")
    FUNCTION = "inp_reference_only"

    CATEGORY = "custom_node_experiments"

    def inp_reference_only(self, model, reference, latent, batch_size):
        model_reference = model.clone()
        size_latent = list(reference["samples"].shape)
        size_latent[0] = batch_size
        # latent = {}
        # latent["samples"] = torch.zeros(size_latent)

        batch = latent["samples"].shape[0] + reference["samples"].shape[0]
        # batch = reference["samples"].shape[0]
        def reference_apply(q, k, v, extra_options):
            k = k.clone().repeat(1, 2, 1)
            offset = 0
            if q.shape[0] > batch:
                offset = batch

            for o in range(0, q.shape[0], batch):
                for x in range(1, batch):
                    k[x + o, q.shape[1]:] = q[o,:]

            return q, k, k

        model_reference.set_model_attn1_patch(reference_apply)
        out_latent = torch.cat((reference["samples"], latent["samples"]))
        if "noise_mask" in latent:
            mask = latent["noise_mask"]
        else:
            mask = torch.ones((64,64), dtype=torch.float32, device="cpu")

        if len(mask.shape) < 3:
            mask = mask.unsqueeze(0)
        if mask.shape[0] < latent["samples"].shape[0]:
            print(latent["samples"].shape, mask.shape)
            mask = mask.repeat(latent["samples"].shape[0], 1, 1)

        out_mask = torch.zeros((1,mask.shape[1],mask.shape[2]), dtype=torch.float32, device="cpu")
        return (model_reference, {"samples": out_latent, "noise_mask": torch.cat((out_mask, mask))})

NODE_CLASS_MAPPINGS = {
    "Inpaint ReferenceOnlySimple": Inp_ReferenceOnlySimple,
}

[addddd2]

@MoonMoon82
Yes, it's essentially the same thing. But it works well for me. I only change the level of noise removal. Values around 0.5 give the original image. Values closer to 0.9 give a reference image

[MoonMoon82]

@addddd2
That just made me curious! I never recognized any similarities to the reference image! So a few minutes ago I tried to reproduce it again. All of a sudden, I realised that all *ancestral-Sampler create the result you mentioned! All non-*ancestral-Sampler just recreate the input image, instead of the reference image at high denoise factor!

Can you confirm my experiences or do you have any additional advices?

[addddd2]

@MoonMoon82 I dont know what is ancestral Sampler, I just use default KSampler
Wokrflows

[MoonMoon82]

@addddd2 I meant the sampler name:

In my experiments, it makes a huge difference! Every other sampler (without "ancestral"-ending of the sampler name) results towards input image instead of the reference image.

Could you please try this combination of sampler+scheduler?

I get error-looking images if I'm using this combination.

[addddd2]

@MoonMoon82
dpmpp_3m_sde_gpu got error

and result

dpmpp_2s_ancestral works fine

[andyjhb]

Traceback (most recent call last):
File "D:\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "D:\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "D:\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "D:\ComfyUI_windows_portable\ComfyUI\custom_nodes\reference_only.py", line 52, in inp_reference_only
return (model_reference, {"samples": out_latent, "noise_mask": torch.cat((out_mask, mask))})
RuntimeError: Tensors must have same number of dimensions: got 3 and 4

[MoonMoon82]

@addddd2 The additional latent input works quite well, but I see at least two issues to solve (which I couldn't solve on my own yet):

• The "batch" number isn't working anymore after our changes. I don't know how to copy latent data n-times.
• I tried to inpaint using "SetLatentNoiseMask", but it ended in an error message. I guess the noise mask tensor shape isn't in the right format. ( Inpainting using Controllnet-Inpaint works fine ;-) )

Do you have an idea how to solve these issues?

[MoonMoon82]

@comfyanonymous You're not going to support this, isn't it? :'(

[addddd2]

@MoonMoon82

Sorry for the long time

inpaint does not work due to the difference in masks sizes

I fixed it like this

--- reference_only.py	2023-07-26 22:24:24.000000000 +0300
+++ reference_only2.py	2023-09-23 15:11:48.232217100 +0300
@@ -1,10 +1,11 @@
 import torch
 
-class ReferenceOnlySimple:
+class ReferenceOnlySimple2:
     @classmethod
     def INPUT_TYPES(s):
         return {"required": { "model": ("MODEL",),
                               "reference": ("LATENT",),
+                              "input2": ("LATENT",),
                               "batch_size": ("INT", {"default": 1, "min": 1, "max": 64})
                               }}
 
@@ -13,12 +14,12 @@
 
     CATEGORY = "custom_node_experiments"
 
-    def reference_only(self, model, reference, batch_size):
+    def reference_only(self, model, reference, input2, batch_size):
         model_reference = model.clone()
         size_latent = list(reference["samples"].shape)
         size_latent[0] = batch_size
-        latent = {}
-        latent["samples"] = torch.zeros(size_latent)
+        latent = input2
+        #latent["samples"] = torch.zeros(size_latent)
 
         batch = latent["samples"].shape[0] + reference["samples"].shape[0]
         def reference_apply(q, k, v, extra_options):
@@ -39,16 +40,22 @@
             mask = latent["noise_mask"]
         else:
             mask = torch.ones((64,64), dtype=torch.float32, device="cpu")
+            
 
         if len(mask.shape) < 3:
             mask = mask.unsqueeze(0)
         if mask.shape[0] < latent["samples"].shape[0]:
             print(latent["samples"].shape, mask.shape)
             mask = mask.repeat(latent["samples"].shape[0], 1, 1)
-
-        out_mask = torch.zeros((1,mask.shape[1],mask.shape[2]), dtype=torch.float32, device="cpu")
+            
+        
+        if len(mask.shape) < 4:
+            out_mask = torch.zeros((1,mask.shape[1],mask.shape[2]), dtype=torch.float32, device="cpu")
+        else:
+            out_mask = torch.zeros((1,1,mask.shape[2],mask.shape[3]), dtype=torch.float32, device="cpu")
+            
         return (model_reference, {"samples": out_latent, "noise_mask": torch.cat((out_mask, mask))})
 
 NODE_CLASS_MAPPINGS = {
-    "ReferenceOnlySimple": ReferenceOnlySimple,
+    "ReferenceOnlySimple2": ReferenceOnlySimple2,
 }

for the batch to work, I suppose some fixes are required in the loops, but I don’t know which ones

And workflow for inpaint I used

[Sullinberg]

HI, I'm new to this stuff and very far away from programming, but I came across this workflows, installed ComfyUI_experiments, but I cant get it - how to install reference_only2.py. Can someone point me in the right direcction?

[Duemellon]

I suspect it's defunct now. I have it installed but can't add more inputs.

[Sullinberg]

@MoonMoon82 @addddd2

I suspect it's defunct now. I have it installed but can't add more inputs.

How do you install the node “reference_only2.py” in experiments I can find only reference_only.py which has only one image input node. Is there a link to file or code to it. Can you share the node and workflow files? Would be very thankfull 😉
Sorry I’m very new to all this but trying to get my way around.