Customized comfyui workflow support.

[huansizhiying]

hope you can follow the example of kritaAI and provide support for customized comfyui workflow in the StableGen plug-in, that is, input images from multiple camera perspectives and some parameters exposed by users, let users build their own workflows to process, and then return the results. Then your plug-in will get the results and continue your other subsequent processes.

[sakalond]

Most of relevant parameters are already exposed within the advanced parameters section. However, I agree that a custom ComfyUI workflow support would be cool.

The problem I see with that is how to integrate said custom workflow into the existing infrastructure of the plugin. One way to do it would be to have fixed prompt, (input image and mask for sequential mode) and fixed output for each viewpoint. Everything in between could in theory be custom. The actual practical implementation for that would be a bit tricky though. Is there anything specific you have in mind which you would custom workflow capability for? Maybe it could just be added as an additional functionality for the plugin if its something useful.

[Iagmon]

Disclaimer: I'm not a dev so I might commit the crime of saying "just do X" when X is overly complicated or time-consuming to do.

I second huansizhiying's suggestion. In my short experience with StableGen, I've caught myself several times thinking things like "How cool would it be if I could use X feature from ComfyUI".

X could be anything, really. Things like Lora weighting per layer, model merging, other ControlNets (OpenPose and others), use of attention masks for ControlNet, the ability to choose to use any ControlNet model I have downloaded, model switching mid-generation, etc.

Some of these may be reasonable additions, but I'm certain others are too niche to be added, so no matter how much features you add, custom workflow support. Let the user create their custom workflows in ComfyUI, save them as .json, load the .json in StableGen and let Comfy run it, with StableGen grabbing the images and bringing them back into Blender.

Of course there would be several problems, such as making the addon's panel compatible with custom workflows. Some of these could be easily solvable by making the user sacrifice convenience for customization by locking in certain parts of the addon's panel. Eg. once the user loads a custom workflow .json, everything under "Core Generation Settings", "Lora Management" and others gets locked up/greyed out/simply disappears, and the only way for the user to, say, change Loras, number of steps or sampler is by modifying his custom .json. Meanwhile, other sections that are relevent only to Blender and not to ComfyUI -such as "Output & Material Settings"- can still be modified/edited on the fly.

The biggest issue I see would be sending the renders from Blender to ComfyUI. I can see this being solved with a "Render camera images and save them in working directory" button, plus a warning for custom workflow users that they need to direct ComfyUI, with their custom workflow, into loading the images from said directory, and that changing the images names or something in the .blend (such as camera or model position) will fuck up the whole process.

[sakalond]

Seems to make sense.

The part with sending Blender renders to ComfyUI is already dealt with, albeit it's not ideal. Currently, it points to the file paths in the set StableGen output directory. For custom workflows, a load image node can be detected and the render's path would be set. Only problem here is to differentiate between different load image nodes within the workflow so user can still do "sequential mode" type of workflow.

As I see it, locking up / greying out most of the settings would be the only realistic way to do it. But yeah, it's probably achievable and even feasible to do. I will add it to the list of planned features.

[huansizhiying]

You can completely rely on kritaAI's custom workflow functionality, a mature solution that's been iterating for two years. You might even consider its existing communication plugin: https://github.com/Acly/comfyui-tooling-nodes.git. I usually create workflows in comfyui, expose parameters, and use them in krita. I believe the same applies to your plugin. Incidentally, I hope to support reading controlnet subfolders soon; it's still problematic.

[VEGETABLET]

Hi! I'm fully into this request and vote for it.

However, I will be completely satisfied if it would be a simple option to select all the necessary models from the list (as it has already been implemented for LoRA selection). You can make a "pro" or smth checkbox for manual vae and clip selection, since I run into some errors in ComfyUI:

`
Failed to validate prompt for output 32:

• ControlNetLoader 201:
  • Value not in list: control_net_name: 'controlnet_flux1_union_pro.safetensors' not in ['Flux\FLUX.1-dev-ControlNet-Union-Pro-2.0.safetensors', 'Flux\Flux_CannyControlNet-v3.safetensors', 'Flux\controlnet_flux1_union_pro.safetensors', 'Flux\flux-canny-controlnet-v3.safetensors', 'Flux\flux-depth-controlnet-v3.safetensors', 'Pony_SDXL\2vxpswa7_OpenposeV2_1.safetensors', 'Pony_SDXL\ControlNet_Union_ProMax_SDXL.safetensors', 'Pony_SDXL\OpenPoseXL2.safetensors', 'Pony_SDXL\illustriousXL_tile_v2.5_controlnet_ep0_step5000.safetensors', 'qwen_image\qwen_image_union_pro.safetensors']
• DualCLIPLoader 11:
  • Value not in list: clip_name1: 't5xxl_fp8_e4m3fn.safetensors' not in ['Flux\fp8_f16\clip_flux_ViT-L-14-TEXT-detail-improved-hiT-GmP-TE-only-HF.safetensors', 'Flux\fp8_f16\google_t5xxl_fp8_e4m3fn.safetensors', 'HiDream\clip_g_hidream.safetensors', 'HiDream\clip_l_hidream.safetensors', 'HiDream\llama_3.1_8b_instruct_fp8_scaled.safetensors', 'Qwen Image\qwen_2.5_vl_7b_fp8_scaled.safetensors', 'SD1.5\SD15_clip.fp16.safetensors', 'Wan 2.1 VACE\umt5_xxl_fp16.safetensors']
  • Value not in list: clip_name2: 'clip_l.safetensors' not in ['Flux\fp8_f16\clip_flux_ViT-L-14-TEXT-detail-improved-hiT-GmP-TE-only-HF.safetensors', 'Flux\fp8_f16\google_t5xxl_fp8_e4m3fn.safetensors', 'HiDream\clip_g_hidream.safetensors', 'HiDream\clip_l_hidream.safetensors', 'HiDream\llama_3.1_8b_instruct_fp8_scaled.safetensors', 'Qwen Image\qwen_2.5_vl_7b_fp8_scaled.safetensors', 'SD1.5\SD15_clip.fp16.safetensors', 'Wan 2.1 VACE\umt5_xxl_fp16.safetensors']
• VAELoader 10:
  • Value not in list: vae_name: 'ae.sft' not in ['Flux\flux_vae.safetensors', 'LTXVideo\LTXV-13B-0.9.8-distilled-VAE.safetensors', 'Pony\sdxl_vae.safetensors', 'Qwen Image\qwen_image_vae.safetensors', 'SD1.5\kl-f8-anime.ckpt', 'SD1.5\klF8Anime2VAE_klF8Anime2VAE.safetensors', 'SDXL\sdxl-vae-fp16-fix.safetensors', 'SDXL\sdxl_vae.safetensors', 'Wan 2.1 VACE\wan_2.1_vae.safetensors', 'Wan 2.2 TI2V-5B\Wan2.2_Quantized_VAE.safetensors', 'Wan 2.2 TI2V-5B\Wan2.2_VAE.pth', 'hunyuan3d-vae-v2-1\model.fp16.ckpt', 'taesd', 'taesdxl', 'taesd3', 'taef1']
    Output will be ignored
    invalid prompt: {'type': 'prompt_outputs_failed_validation', 'message': 'Prompt outputs failed validation', 'details': '', 'extra_info': {}}
    `

As you can see, I have all the necessary models, but it's names are different or they placed in the different folders!

Regarding the custom ComfyUI workflow support: stableprojectorz.com has such an option, so you can use this as reference. Basically, it has a minimum package for just prompting and 3d stuff and custom ComfyUI nodes for input/output from/to application. I find the interface and operations of the StableProjectorz inconvenient and quite awkward, so I would like to see your blender addon succeeded! Good luck in development!

[sakalond]

A quick reaction to #43:

Yes, that's true. It isn't really a problem afterall. Yet doing custom ComfyUI workflows is still tricky in regard to the UX. I don't really have a clear idea of how this should be implemented when there is so much "hardwired" logic which has to be done for the texture generation to work at all. I don't see much space for anything more custom than what is already being offered through the already pretty extensive advanced parameters section.

[sakalond]

However, I will be completely satisfied if it would be a simple option to select all the necessary models from the list (as it has already been implemented for LoRA selection). You can make a "pro" or smth checkbox for manual vae and clip selection, since I run into some errors in ComfyUI:

Custom VAE and clip selection should not be an issue. I will implement it soon.